Mixture Densities, Maximum Likelihood and the Em Algorithm

• Published in: SIAM review Vol. 26; no. 2; pp. 195 - 239
• Main Authors: Redner, Richard A., Walker, Homer F.
• Format: Journal Article
• Language: English
• Published: Philadelphia, PA The Society for Industrial and Applied Mathematics 01.04.1984; Society for Industrial and Applied Mathematics
• Subjects: Algorithms; Consistent estimators; Density; Density estimation; Estimates; Estimation methods; Exact sciences and technology; Gaussian distributions; Mathematical independent variables; Mathematics; Maximum likelihood estimation; Parametric inference; Perceptron convergence procedure; Probability; Probability and statistics; Sciences and techniques of general use; Statistics; Probability density; Missing data; Exponential distribution; Statistical estimation; Maximum likelihood; Mixed probability; Algorithm
• ISSN: 0036-1445; 1095-7200
• DOI: 10.1137/1026034

The problem of estimating the parameters which determine a mixture density has been the subject of a large, diverse body of literature spanning nearly ninety years. During the last two decades, the method of maximum likelihood has become the most widely followed approach to this problem, thanks primarily to the advent of high speed electronic computers. Here, we first offer a brief survey of the literature directed toward this problem and review maximum-likelihood estimation for it. We then turn to the subject of ultimate interest, which is a particular iterative procedure for numerically approximating maximum-likelihood estimates for mixture density problems. This procedure, known as the EM algorithm, is a specialization to the mixture density context of a general algorithm of the same name used to approximate maximum-likelihood estimates for incomplete data problems. We discuss the formulation and theoretical and practical properties of the EM algorithm for mixture densities, focussing in particular on mixtures of densities from exponential families.