Convergence Properties of High-order Boltzmann Machines

• Published in: Neural networks Vol. 9; no. 9; pp. 1561 - 1567
• Main Authors: Albizuri, F.Xabier, d'Anjou, Alicia, Graña, Manuel, Lozano, J.Antonio
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.12.1996; Elsevier Science
• Subjects: Applied sciences; Approximation of probability distributions; Artificial intelligence; Boltzmann machine; Computer science; control theory; systems; Connectionism. Neural networks; Exact sciences and technology; High-order neural networks; Learning algorithms; Maximum likelihood estimates; Monte Carlo methods; Statistical inference; Stochastic neural networks; High-order neural networks; Learning algorithms; Monte Carlo methods; Boltzmann machine; Stochastic neural networks; Approximation of probability distributions; Statistical inference; Maximum likelihood estimates; Learning; Monte Carlo method; Approximation; Probability distribution; Statistical decision; Maximum likelihood; Neural network; Algorithm; Modeling; Stochastic system
• ISSN: 0893-6080; 1879-2782
• DOI: 10.1016/S0893-6080(96)00026-3

The high-order Boltzmann machine (HOBM) approximates probability distributions defined on a set of binary variables, through a learning algorithm that uses Monte Carlo methods. The approximation distribution is a normalized exponential of a consensus function formed by high-degree terms and the structure of the HOBM is given by the set of weighted connections. We prove the convexity of the Kullback-Leibler divergence between the distribution to learn and the approximation distribution of the HOBM. We prove the convergence of the learning algorithm to the strict global minimum of the divergence, which corresponds to the maximum likelihood estimate of the connection weights, establishing the uniqueness of the solution. These theoretical results do not hold in the conventional Boltzmann machine, where the consensus function has first and second-degree terms and hidden units are used. Copyright © 1996 Elsevier Science Ltd.