Universal Source Coding for Monotonic and Fast Decaying Monotonic Distributions

• Published in: IEEE transactions on information theory Vol. 59; no. 11; pp. 7194 - 7211
• Main Author: Shamir, Gil I.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.11.2013; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Average redundancy; Codes; Coding, codes; Decoding; Encoding; Entropy; Exact sciences and technology; individual redundancy; Information theory; Information, signal and communications theory; large alphabets; monotonic distributions; patterns; Probability distribution; Quantization (signal); Redundancy; Signal and communications theory; Telecommunications and information theory; universal compression; Upper bound; Alphabet; Universal coding; large alphabets; monotonic distributions; Source coding; Redundancy; individual redundancy; patterns; Entropy; universal compression; Geometric distribution; Average redundancy
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2013.2281695

We study universal compression of sequences generated by monotonic distributions. We show that for a monotonic distribution over an alphabet of size k, each probability parameter costs essentially 0.5log(n/k 3 ) bits, where n is the coded sequence length, as long as k=o(n 1/3 ). Otherwise, for k=O(n), the total average sequence redundancy is O(n 1/3+ε ) bits overall. We then show that there exists a sub-class of monotonic distributions over infinite alphabets for which redundancy of O(n 1/3+ε ) bits overall is still achievable. This class contains fast decaying distributions, including many distributions over the integers such as the family of Zipf distributions and geometric distributions. For some slower decays, including other distributions over the integers, redundancy of o(n) bits overall is achievable. A method to compute specific redundancy rates for such distributions is derived. The results are specifically true for finite entropy monotonic distributions. Finally, we study individual sequence redundancy behavior assuming a sequence is governed by a monotonic distribution. We show that for sequences whose empirical distributions are monotonic, individual redundancy bounds even tighter than those in the average case can be obtained. The relation of universal compression with monotonic distributions to universal compression of patterns of sequences is demonstrated.