On minimizing distortion and relative entropy

• Published in: IEEE transactions on information theory Vol. 52; no. 1; pp. 238 - 245
• Main Authors: Friedlander, M.P., Gupta, M.R.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.01.2006; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Computer science; Constraint optimization; Convex optimization; Councils; cross-entropy; Decay; Distortion; Electrical engineering; Entropy; Equations; Estimating; exact penalty; Exact sciences and technology; Formulations; function; Information theory; Information, signal and communications theory; inverse problem; Inverse problems; Kullback-Leibler distance; Mathematical functions; maximum entropy; moment constraint; Nonlinear distortion; Norms; Power engineering and energy; Probability; Random variables; relative entropy; Scientific computing; Telecommunications and information theory; Kullback-Leibler distance; cross-entropy; maximum entropy; Method of maximum entropy; Convex programming; Optimization; Inverse problem; Mass function; exact penalty; Convex optimization; moment constraint; function; relative entropy; Information theory
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2005.860448

A common approach for estimating a probability mass function w when given a prior q and moment constraints given by Aw/spl les/b is to minimize the relative entropy between w and q subject to the set of linear constraints. In such cases, the solution w is known to have exponential form. We consider the case in which the linear constraints are noisy, uncertain, infeasible, or otherwise "soft." A solution can then be obtained by minimizing both the relative entropy and violation of the constraints Aw/spl les/b. A penalty parameter /spl sigma/ weights the relative importance of these two objectives. We show that this penalty formulation also yields a solution w with exponential form. If the distortion is based on an /spl lscr//sub p/ norm, then the exponential form of w is shown to have exponential decay parameters that are bounded as a function of /spl sigma/. We also state conditions under which the solution w to the penalty formulation will result in zero distortion, so that the moment constraints hold exactly. These properties are useful in choosing penalty parameters, evaluating the impact of chosen penalty parameters, and proving properties about methods that use such penalty formulations.