Meta Derivative Identity for the Conditional Expectation

• Published in: IEEE transactions on information theory Vol. 69; no. 7; pp. 4284 - 4302
• Main Authors: Dytso, Alex, Cardone, Martina, Zieder, Ian
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Bayesian risk; conditional cumulants; Conditional mean estimator; Cramér-Rao bound; Estimation error; exponential family; Gaussian noise; Gibbs distribution; Identities; information density; Jacobi matrix method; Jacobian matrices; Jacobian matrix; log-Sobolev inequality (LSI); Lower bounds; Markov chains; Markov processes; minimum mean squared error (MMSE); Operators (mathematics); Poincaré inequality; Probability density function; Random noise; Random variables; score function; Signal processing; Tweedie’s formula
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2023.3249163

Consider a pair of random vectors <inline-formula> <tex-math notation="LaTeX">( {\mathbf{X}}, {\mathbf{Y}}) </tex-math></inline-formula> and the conditional expectation operator <inline-formula> <tex-math notation="LaTeX"> \mathbb {E}[ {\mathbf{X}}| {\mathbf{Y}}={\mathbf{y}}] </tex-math></inline-formula>. This work studies analytical properties of the conditional expectation by characterizing various derivative identities. The paper consists of two parts. In the first part of the paper, a general derivative identity for the conditional expectation is derived. Specifically, for the Markov chain <inline-formula> <tex-math notation="LaTeX"> {\mathbf{U}}\leftrightarrow {\mathbf{X}}\leftrightarrow {\mathbf{Y}} </tex-math></inline-formula>, a compact expression for the Jacobian matrix of <inline-formula> <tex-math notation="LaTeX"> \mathbb {E}[ \psi ( {\mathbf{Y}}, {\mathbf{U}})| {\mathbf{Y}}= {\mathbf{y}}] </tex-math></inline-formula> for a smooth function <inline-formula> <tex-math notation="LaTeX">\psi </tex-math></inline-formula> is derived. In the second part of the paper, the main identity is specialized to the exponential family and two main applications are shown. First, it is demonstrated that, via various choices of the random vector <inline-formula> <tex-math notation="LaTeX"> {\mathbf{U}} </tex-math></inline-formula> and function <inline-formula> <tex-math notation="LaTeX">\psi </tex-math></inline-formula>, one can recover and generalize several known identities (e.g., Tweedie's formula) and derive some new ones. For example, a new relationship between conditional expectations and conditional cumulants is established. Second, it is demonstrated how the derivative identities can be used to establish new lower bounds on the estimation error. More specifically, using one of the derivative identities in conjunction with a Poincaré inequality, a new lower bound on the minimum mean squared error, which holds for all prior distributions on the input signal, is derived. The new lower bound is shown to be tight in the high-noise regime for the additive Gaussian noise setting.