Compressing and interpreting word embeddings with latent space regularization and interactive semantics probing

• Published in: Information visualization Vol. 22; no. 1; pp. 52 - 68
• Main Authors: Li, Haoyu, Wang, Junpeng, Zheng, Yan, Wang, Liang, Zhang, Wei, Shen, Han-Wei
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.01.2023; SAGE PUBLICATIONS, INC
• Subjects: Compressing; Embedding; Machine learning; Natural language processing; Photovoltaic cells; Regularization; Representations; Semantics; Words (language); visual analytics; word embedding; High-dimensional data visualization; neural networks
• DOI: 10.1177/14738716221130338
• ISSN: 1473-8716

Word embedding, a high-dimensional (HD) numerical representation of words generated by machine learning models, has been used for different natural language processing tasks, for example, translation between two languages. Recently, there has been an increasing trend of transforming the HD embeddings into a latent space (e.g. via autoencoders) for further tasks, exploiting various merits the latent representations could bring. To preserve the embeddings’ quality, these works often map the embeddings into an even higher-dimensional latent space, making the already complicated embeddings even less interpretable and consuming more storage space. In this work, we borrow the idea of β VAE to regularize the HD latent space. Our regularization implicitly condenses information from the HD latent space into a much lower-dimensional space, thus compressing the embeddings. We also show that each dimension of our regularized latent space is more semantically salient, and validate our assertion by interactively probing the encoding-level of user-proposed semantics in the dimensions. To the end, we design a visual analytics system to monitor the regularization process, explore the HD latent space, and interpret latent dimensions’ semantics. We validate the effectiveness of our embedding regularization and interpretation approach through both quantitative and qualitative evaluations.