Bayesian Inference Featuring Entropic Priors

• Published in: Bayesian Inference and Maximum Entropy Methods in Science and Engineering (AIP Conference Proceedings Volume 954) Vol. 954; pp. 283 - 292
• Main Author: Neumann, Tilman
• Format: Journal Article
• Language: English
• Published: 01.01.2007
• ISBN: 0735404682; 9780735404687
• ISSN: 0094-243X
• DOI: 10.1063/1.2821274

The subject of this work is the parametric inference problem, i.e. how to infer from data on the parameters of the data likelihood of a random process whose parametric form is known a priori. The assumption that Bayes'theorem has to be used to add new data samples reduces the problem to the question of how to specify a prior before having seen any data. For this subproblem three theorems are stated. The first one is that Jaynes' Maximum Entropy Principle requires at least a constraint on the expected data likelihood entropy, which gives entropic priors without the need of further axioms. Second I show that maximizing Shannon entropy under an expected data likelihood entropy constraint is equivalent to maximizing relative entropy and therefore reparametrization invariant for continuous-valued data likelihoods. Third, I propose that in the state of absolute ignorance of the data likelihood entropy, one should choose the hyperparameter alpha of an entropic prior such that the change of expected data likelihood entropy is maximized. Among other beautiful properties, this principle is equivalent to the maximization of the mean-squared entropy error and invariant against any reparametrizations of the data likelihood. Altogether we get a Bayesian inference procedure that incorporates special prior knowledge if available but has also a sound solution if not, and leaves no hyperparameters unspecified.