A Distributional Semantic Online Lexicon for Linguistic Explorations of Societies

• Published in: Social science computer review Vol. 41; no. 2; pp. 308 - 329
• Main Authors: Dahlberg, Stefan, Axelsson, Sofia, Gyllensten, Amaru Cuba, Sahlgren, Magnus, Ekgren, Ariel, Holmberg, Sören, Schwarz, Jonas Andersson
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.04.2023; SAGE PUBLICATIONS, INC
• Subjects: Annan samhällsvetenskap; comparative surveys; Computer Science; Data; Data processing; democracy; distributional semantics; Information Science & Library Science; Interdisciplinary aspects; Interdisciplinary studies; Internet; Language Technology (Computational Linguistics); language use; Languages; Linguistics; Natural language processing; Other Social Sciences; Other Topics; Political behavior; Political science; Polls & surveys; Public opinion; Research projects; satisfaction; semantic similarities; Semantics; Social Sciences; Språkteknologi (språkvetenskaplig databehandling); Translations; word2vec; natural language processing; semantic similarities; word2vec; comparative surveys; language use; distributional semantics
• ISSN: 0894-4393; 1552-8286; 1552-8286
• DOI: 10.1177/08944393211049774

Linguistic Explorations of Societies (LES) is an interdisciplinary research project with scholars from the fields of political science, computer science, and computational linguistics. The overarching ambition of LES has been to contribute to the survey-based comparative scholarship by compiling and analyzing online text data within and between languages and countries. To this end, the project has developed an online semantic lexicon, which allows researchers to explore meanings and usages of words in online media across a substantial number of geo-coded languages. The lexicon covers data from approximately 140 language–country combinations and is, to our knowledge, the most extensive free research resource of its kind. Such a resource makes it possible to critically examine survey translations and identify discrepancies in order to modify and improve existing survey methodology, and its unique features further enable Internet researchers to study public debate online from a comparative perspective. In this article, we discuss the social scientific rationale for using online text data as a complement to survey data, and present the natural language processing–based methodology behind the lexicon including its underpinning theory and practical modeling. Finally, we engage in a critical reflection about the challenges of using online text data to gauge public opinion and political behavior across the world.