Entropy of Patterns of i.i.d. Sequences-Part I: General Bounds

• Published in: IEEE transactions on information theory Vol. 54; no. 5; pp. 2263 - 2277
• Main Author: Shamir, G.I.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.05.2008; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Approximation; Cities and towns; Communication system control; Conferences; Data processing; Dictionaries; Entropy; Exact sciences and technology; Gas insulated transmission lines; index sequences; Information theory; Information, signal and communications theory; Pattern matching; patterns; Probability; Symbols; Telecommunications and information theory; Uncertainty; Alphabet; Lower bound; Upper bound; index sequences; patterns; Data processing; Entropy
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2008.920203

Tight bounds on the block entropy of patterns of sequences generated by independent and identically distributed (i.i.d.) sources are derived. A pattern of a sequence is a sequence of integer indices with each index representing the order of first occurrence of the respective symbol in the original sequence. Since a pattern is the result of data processing on the original sequence, its entropy cannot be larger. Bounds derived here describe the pattern entropy as function of the original i.i.d. source entropy, the alphabet size, the symbol probabilities, and their arrangement in the probability space. Matching upper and lower bounds derived provide a useful tool for very accurate approximations of pattern block entropies for various distributions, and for assessing the decrease of the pattern entropy from that of the original i.i.d. sequence.