Functional Bregman Divergence and Bayesian Estimation of Distributions

• Published in: IEEE transactions on information theory Vol. 54; no. 11; pp. 5130 - 5139
• Main Authors: Frigyik, B.A., Srivastava, S., Gupta, M.R.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.11.2008; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Bayesian analysis; Bayesian estimation; Bayesian methods; Bregman divergence; convexity; Data processing; Distortion; Distortion measurement; Divergence; Entropy; Estimating techniques; Estimation theory; Exact sciences and technology; FrÉchet derivative; Information theory; Information, signal and communications theory; Inverse problems; Logistics; Mathematical analysis; Mathematical functions; Mathematics; Statistical learning; Telecommunications and information theory; Theorems; uniform distribution; Vectors (mathematics); Bayes estimation; Case study; Uniform distribution; Bayesian estimation; Fréchet derivative; Bregman divergence; Entropy; Convexity
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2008.929943

A class of distortions termed functional Bregman divergences is defined, which includes squared error and relative entropy. A functional Bregman divergence acts on functions or distributions, and generalizes the standard Bregman divergence for vectors and a previous pointwise Bregman divergence that was defined for functions. A recent result showed that the mean minimizes the expected Bregman divergence. The new functional definition enables the extension of this result to the continuous case to show that the mean minimizes the expected functional Bregman divergence over a set of functions or distributions. It is shown how this theorem applies to the Bayesian estimation of distributions. Estimation of the uniform distribution from independent and identically drawn samples is presented as a case study.