Information rate of meaningful communication

• Published in: Proceedings of the National Academy of Sciences - PNAS Vol. 122; no. 25; p. e2502353122
• Main Authors: Sivan, Doron, Tsodyks, Misha
• Format: Journal Article
• Language: English
• Published: United States National Academy of Sciences 24.06.2025
• Subjects: Large language models; Physical Sciences; Stochastic processes; large language models; information theory; semantics

In Shannon’s seminal paper, the entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: i) the units of information are not character...