On minimizing distortion and relative entropy

• Published in: IEEE transactions on information theory Vol. 52; no. 1; pp. 238 - 245
• Main Authors: Friedlander, M.P., Gupta, M.R.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.01.2006; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Computer science; Constraint optimization; Convex optimization; Councils; cross-entropy; Decay; Distortion; Electrical engineering; Entropy; Equations; Estimating; exact penalty; Exact sciences and technology; Formulations; function; Information theory; Information, signal and communications theory; inverse problem; Inverse problems; Kullback-Leibler distance; Mathematical functions; maximum entropy; moment constraint; Nonlinear distortion; Norms; Power engineering and energy; Probability; Random variables; relative entropy; Scientific computing; Telecommunications and information theory; Kullback-Leibler distance; cross-entropy; maximum entropy; Method of maximum entropy; Convex programming; Optimization; Inverse problem; Mass function; exact penalty; Convex optimization; moment constraint; function; relative entropy; Information theory

A common approach for estimating a probability mass function w when given a prior q and moment constraints given by Aw/spl les/b is to minimize the relative entropy between w and q subject to the set of linear constraints. In such cases, the solution w is known to have exponential form. We consider...