Generalized Permutation Framework for Testing Model Variable Significance

A common problem in machine learning is determining if a variable significantly contributes to a model's prediction performance. This problem is aggravated for datasets, such as gene expression datasets, that suffer the worst case of dimensionality: a low number of observations along with a hig...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Wu, Yue, Spaide, Ted, Nakamichi, Kenji, Russell Van Gelder, Lee, Aaron
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 12.10.2023
Subjects
Confidence intervals
Datasets
Gene expression
Machine learning
Model testing
Permutations
Statistical analysis
Statistical methods
Variables
Online AccessGet full text

Cover

More Information
Summary:A common problem in machine learning is determining if a variable significantly contributes to a model's prediction performance. This problem is aggravated for datasets, such as gene expression datasets, that suffer the worst case of dimensionality: a low number of observations along with a high number of possible explanatory variables. In such scenarios, traditional methods for testing variable statistical significance or constructing variable confidence intervals do not apply. To address these problems, we developed a novel permutation framework for testing the significance of variables in supervised models. Our permutation framework has three main advantages. First, it is non-parametric and does not rely on distributional assumptions or asymptotic results. Second, it not only ranks model variables in terms of relative importance, but also tests for statistical significance of each variable. Third, it can test for the significance of the interaction between model variables. We applied this permutation framework to multi-class classification of the Iris flower dataset and of brain regions in RNA expression data, and using this framework showed variable-level statistical significance and interactions.
ISSN:2331-8422