Controlled Abstention Neural Networks for Identifying Skillful Predictions for Classification Problems

• Published in: Journal of advances in modeling earth systems Vol. 13; no. 12
• Main Authors: Barnes, Elizabeth A., Barnes, Randal J.
• Format: Journal Article
• Language: English
• Published: Washington John Wiley & Sons, Inc 01.12.2021
• Subjects: Classification; Confidence; Datasets; Deep learning; El Nino; forecasts of opportunity; Learning behaviour; Machine learning; Neural networks; Normal distribution; prediction; Training; uncertainty; Weather forecasting
• ISSN: 1942-2466; 1942-2466
• DOI: 10.1029/2021MS002573

The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed “forecasts of opportunity.” When these opportunities are not present, scientists need prediction systems that are capable of saying “I don't know.” We introduce a novel loss function, termed the “NotWrong loss,” that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss introduces an abstention class that allows the network to identify the more confident samples and abstain (say “I don't know”) on the less confident samples. The abstention loss is designed to abstain on a user‐defined fraction of the samples via a standard adaptive controller. Unlike many machine learning methods used to reject samples post‐training, the NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases. The implementation of the proposed loss function is straightforward in most network architectures designed for classification as it only requires the addition of an abstention class to the output layer and modification of the loss function. Plain Language Summary The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we can look for specific states of the system that lead to more predictable behavior than others, often termed “forecasts of opportunity”. When these opportunities are not present, scientists need prediction systems that are capable of saying “I don't know.” We present a method for teaching neural networks, a type of machine learning tool, to say “I don't know” for classification problems. By doing so, the neural network focuses less on the predictions it identifies as problematic and focuses more on the predictions where its confidence is high. In the end, this leads to better predictions. Key Points A simple neural network approach for abstention is explored for climate classification problems A new abstention loss is introduced to identify, and preferentially learn from, more confident samples This new abstention loss improves prediction accuracy for a variety of climate use cases