Extrinsic Jensen-Shannon Divergence: Applications to Variable-Length Coding

• Published in: IEEE transactions on information theory Vol. 61; no. 4; pp. 2148 - 2164
• Main Authors: Naghshvar, Mohammad, Javidi, Tara, Wigger, Michele
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Burnashev's reliability function; Channel coding; Coding theory; Computer Science; Decoding; Discrete memoryless channel; feedback gain; Information Theory; Noise; optimal error exponent; Receivers; sequential analysis; Stochastic models; Transmitters; Uncertainty; Upper bound; variable-length coding; optimal error exponent; Discrete memoryless channel; sequential analysis; variable-length coding; Burnashev’s reliability function; feedback gain
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2015.2401004

This paper considers the problem of variable-length coding over a discrete memoryless channel with noiseless feedback. This paper provides a stochastic control view of the problem whose solution is analyzed via a newly proposed symmetrized divergence, termed extrinsic Jensen-Shannon (EJS) divergence. It is shown that strictly positive lower bounds on EJS divergence provide nonasymptotic upper bounds on the expected code length. This paper presents strictly positive lower bounds on EJS divergence, and hence nonasymptotic upper bounds on the expected code length, for the following two coding schemes: 1) variable-length posterior matching and 2) MaxEJS coding scheme that is based on a greedy maximization of the EJS divergence. As an asymptotic corollary of the main results, this paper also provides a rate-reliability test. Variable-length coding schemes that satisfy the condition(s) of the test for parameters R and E are guaranteed to achieve a rate R and an error exponent E. The results are specialized for posterior matching and MaxEJS to obtain deterministic one-phase coding schemes achieving capacity and optimal error exponent. For the special case of symmetric binary-input channels, simpler deterministic schemes of optimal performance are proposed and analyzed.