Maximum Rényi Entropy Rate

• Published in: IEEE transactions on information theory Vol. 62; no. 3; pp. 1193 - 1205
• Main Authors: Bunte, Christoph, Lapidoth, Amos
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Burg's Theorem; Computation; Density measurement; Electronic mail; Entropy; Entropy (Information); entropy rate; Indexes; Information theory; Logarithms; Mathematical analysis; Mathematical models; maximization; Normal distribution; Probability density function; Renyi entropy; Renyi entropy rate; spectrum estimation; Stochastic models; Stochastic processes; Upper bound; Vectors (mathematics); maximization; Rényi entropy rate; spectrum estimation; Rényi entropy; entropy rate; Burg’s Theorem
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2016.2521364

The supremum of the Rényi entropy rate over the class of discrete-time stationary stochastic processes, whose marginals are supported by some given set and satisfy some given cost constraint, is computed. Unlike the Shannon entropy, the Rényi entropy of a random vector can exceed the sum of the Rényi entropies of its components, and the supremum is, therefore, typically not achieved by memoryless processes. It is nonetheless related to Shannon's entropy: when the Rényi parameter exceeds one, the supremum is equal to the corresponding supremum of Shannon's entropy, and when it is smaller than one, the supremum equals the logarithm of the volume of the support set. A Burg-like supremum of the Rényi entropy rate over the class of stochastic processes, whose autocovariance function begins with some given values, is also solved. It is not achieved by Gauss-Markov processes, but it is nonetheless related to Burg's supremum: the two are equal when the Rényi parameter exceeds one, and the former is infinite otherwise.