On the Dimensionality of Embeddings for Sparse Features and Data

In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent spa...

Full description

Saved in:
Bibliographic Details
Main Author Naumov, Maxim
Format Journal Article
LanguageEnglish
Published 07.01.2019
Subjects
Computer Science - Computer Vision and Pattern Recognition
Computer Science - Information Theory
Computer Science - Learning
Mathematics - Information Theory
Statistics - Machine Learning
Online AccessGet full text

Cover

More Information
Summary:In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent sparse features or data, may or not reduce the dimensionality of the item space. However, the embeddings do provide a different and often more meaningful representation of the items for a particular task at hand. Also, we give upper bounds and more precise guidelines for choosing the embedding dimension.
DOI:10.48550/arxiv.1901.02103