Estimating Entropy Rates with Bayesian Confidence Intervals

• Published in: Neural computation Vol. 17; no. 7; pp. 1531 - 1576
• Main Authors: Kennel, Matthew B., Shlens, Jonathon, Abarbanel, Henry D. I., Chichilnisky, E. J.
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.07.2005; MIT Press Journals, The
• Subjects: Applied sciences; Artificial intelligence; Bayes Theorem; Biological and medical sciences; Computer science; control theory; systems; Confidence Intervals; Entropy; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects. Models. Methods; Learning and adaptive systems; Mathematics; Neural networks; Neurons; Nonlinear dynamics and nonlinear dynamical systems; Parametric inference; Physics; Probability and statistics; Sciences and techniques of general use; Statistical physics, thermodynamics, and nonlinear dynamical systems; Statistics; Statistics as Topic; Vertebrates: nervous system and sense organs; Bayes estimation; Monte Carlo method; Spiking neuron; Neural computation; Uncertainty; Experimental data; Balancing; Disordered system; Model selection; Data compression; Bias; Entropy; Statistical estimation; Neural network; Dynamical system; Confidence interval; Weighting; Simulation; Minimum principle; Observation data; Selection method; Probabilistic model; Biased estimation
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/0899766053723050

The entropy rate quantifies the amount of uncertainty or disorder produced by any dynamical system. In a spiking neuron, this uncertainty translates into the amount of information potentially encoded and thus the subject of intense theoretical and experimental investigation. Estimating this quantity in observed, experimental data is difficult and requires a judicious selection of probabilistic models, balancing between two opposing biases. We use a model weighting principle originally developed for lossless data compression, following the minimum description length principle. This weighting yields a direct estimator of the entropy rate, which, compared to existing methods, exhibits significantly less bias and converges faster in simulation. With Monte Carlo techinques, we estimate a Bayesian confidence interval for the entropy rate. In related work, weap-ply these ideas to estimate the information rates between sensory stimuli and neural responses in experimental data (Shlens, Kennel, Abarbanel, & Chichilnisky, in preparation).