The Variational Gaussian Approximation Revisited

• Published in: Neural computation Vol. 21; no. 3; pp. 786 - 792
• Main Authors: Opper, Manfred, Archambeau, Cédric
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.03.2009; MIT Press Journals, The
• Subjects: Applied sciences; Approximation; Artificial Intelligence; Biological and medical sciences; Computer science; control theory; systems; Distribution theory; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; Humans; Learning and adaptive systems; Letters; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Miscellaneous; Normal Distribution; Parameter optimization; Probability and statistics; Random variables; Regression analysis; Sciences and techniques of general use; Statistics; Normal approximation; Neural computation; Prior distribution; Gaussian distribution; Multivariate distribution; Neural network; Multivariate analysis; Random number; Statistical regression; Gaussian process; Gaussian processes; Distribution function; Random variable; Posterior distribution; Laplace approximation
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/neco.2008.08-07-592

The variational approximation of posterior distributions by multivariate gaussians has been much less popular in the machine learning community compared to the corresponding approximation by factorizing distributions. This is for a good reason: the gaussian approximation is in general plagued by an number of variational parameters to be optimized, being the number of random variables. In this letter, we discuss the relationship between the Laplace and the variational approximation, and we show that for models with gaussian priors and factorizing likelihoods, the number of variational parameters is actually . The approach is applied to gaussian process regression with nongaussian likelihoods.