Learning to Explain: A Model-Agnostic Framework for Explaining Black Box Models

• Published in: 2023 IEEE International Conference on Data Mining (ICDM) pp. 944 - 949
• Main Authors: Barkan, Oren, Asher, Yuval, Eshel, Amit, Elisha, Yehonatan, Koenigstein, Noam
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2023
• Subjects: Closed box; computer vision; Explainable AI; Measurement; Natural language processing; Optimization; Predictive models; Training; Transformers
• ISSN: 2374-8486
• DOI: 10.1109/ICDM58522.2023.00105

We present Learning to Explain (LTX), a model-agnostic framework designed for providing post-hoc explanations for vision models. The LTX framework introduces an "explainer" model that generates explanation maps, highlighting the crucial regions that justify the predictions made by the model being explained. To train the explainer, we employ a two-stage process consisting of initial pretraining followed by per-instance finetuning. During both stages of training, we utilize a unique configuration where we compare the explained model's prediction for a masked input with its original prediction for the unmasked input. This approach enables the use of a novel counterfactual objective, which aims to anticipate the model's output using masked versions of the input image. Importantly, the LTX framework is not restricted to a specific model architecture and can provide explanations for both Transformer-based and convolutional models. Through our evaluations, we demonstrate that LTX significantly outperforms the current state-of-the-art in explainability across various metrics. Our code is available at: https://github.comLTX-CodeLTX