Large Scale Substitution-based Word Sense Induction

We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Matan Eyal, Shoval Sadde, Taub-Tabib, Hillel, Goldberg, Yoav
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 21.03.2022
Subjects
Algorithms
Data analysis
Datasets
Outliers (statistics)
Online AccessGet full text

Cover

More Information
Summary:We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy. Furthermore, by training a static word embeddings algorithm on the sense-tagged corpus, we obtain high-quality static senseful embeddings. These outperform existing senseful embeddings methods on the WiC dataset and on a new outlier detection dataset we developed. The data driven nature of the algorithm allows to induce corpora-specific senses, which may not appear in standard sense inventories, as we demonstrate using a case study on the scientific domain.
ISSN:2331-8422