Ratio Divergence Learning Using Target Energy in Restricted Boltzmann Machines: Beyond Kullback--Leibler Divergence Learning

• Main Authors: Ishida, Yuichi, Ichikawa, Yuma, Dote, Aki, Miyazawa, Toshiyuki, Hukushima, Koji
• Format: Journal Article
• Language: English
• Published: 11.09.2024
• Subjects: Computer Science - Learning; Mathematics - Statistics Theory; Physics - Disordered Systems and Neural Networks; Statistics - Machine Learning; Statistics - Methodology; Statistics - Theory
• DOI: 10.48550/arxiv.2409.07679

We propose ratio divergence (RD) learning for discrete energy-based models, a method that utilizes both training data and a tractable target energy function. We apply RD learning to restricted Boltzmann machines (RBMs), which are a minimal model that satisfies the universal approximation theorem for discrete distributions. RD learning combines the strength of both forward and reverse Kullback-Leibler divergence (KLD) learning, effectively addressing the "notorious" issues of underfitting with the forward KLD and mode-collapse with the reverse KLD. Since the summation of forward and reverse KLD seems to be sufficient to combine the strength of both approaches, we include this learning method as a direct baseline in numerical experiments to evaluate its effectiveness. Numerical experiments demonstrate that RD learning significantly outperforms other learning methods in terms of energy function fitting, mode-covering, and learning stability across various discrete energy-based models. Moreover, the performance gaps between RD learning and the other learning methods become more pronounced as the dimensions of target models increase.