Compositional Evaluation on Japanese Textual Entailment and Similarity

• Published in: Transactions of the Association for Computational Linguistics Vol. 10; pp. 1266 - 1284
• Main Authors: Yanaka, Hitomi, Mineshima, Koji
• Format: Journal Article
• Language: English
• Published: One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA MIT Press 22.11.2022; MIT Press Journals, The; The MIT Press
• Subjects: Case marking; Datasets; English language; Entailment; Function words; Grammatical case; Inference; Japanese language; Language; Language modeling; Language universals; Multilingualism; Natural language processing; Semantics; Sentence structure; Similarity; Stress; Syntactic structures; Syntax; Word order; Words (language)
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00518

Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.