Embedding Learning

• Published in: Journal of the American Statistical Association Vol. 117; no. 537; pp. 307 - 319
• Main Authors: Dai, Ben, Shen, Xiaotong, Wang, Junhui
• Format: Journal Article
• Language: English
• Published: United States Taylor & Francis 02.01.2022; Taylor & Francis Ltd
• Subjects: Algorithms; Americans; Automatic feature generation; Classification; Classifiers; Coders; data analysis; Data mining; Deep learning; dimensions; Embedding; Graphical representations; graphs; Learning; Learning theory; Mapping; Mathematical analysis; Natural language processing; Neural networks; Performance prediction; Regression analysis; Representational learning; Sentiment analysis; Sentimental analysis; Statistical methods; Statistics; Tensors; Text mining; Unstructured data; Deep learning; Representational learning; Text mining; Natural language processing; Sentimental analysis; Automatic feature generation
• ISSN: 0162-1459; 1537-274X; 1537-274X
• DOI: 10.1080/01621459.2020.1775614

Numerical embedding has become one standard technique for processing and analyzing unstructured data that cannot be expressed in a predefined fashion. It stores the main characteristics of data by mapping it onto a numerical vector. An embedding is often unsupervised and constructed by transfer learning from large-scale unannotated data. Given an embedding, a downstream learning method, referred to as a two-stage method, is applicable to unstructured data. In this article, we introduce a novel framework of embedding learning to deliver a higher learning accuracy than the two-stage method while identifying an optimal learning-adaptive embedding. In particular, we propose a concept of U-minimal sufficient learning-adaptive embeddings, based on which we seek an optimal one to maximize the learning accuracy subject to an embedding constraint. Moreover, when specializing the general framework to classification, we derive a graph embedding classifier based on a hyperlink tensor representing multiple hypergraphs, directed or undirected, characterizing multi-way relations of unstructured data. Numerically, we design algorithms based on blockwise coordinate descent and projected gradient descent to implement linear and feed-forward neural network classifiers, respectively. Theoretically, we establish a learning theory to quantify the generalization error of the proposed method. Moreover, we show, in linear regression, that the one-hot encoder is more preferable among two-stage methods, yet its dimension restriction hinders its predictive performance. For a graph embedding classifier, the generalization error matches up to the standard fast rate or the parametric rate for linear or nonlinear classification. Finally, we demonstrate the utility of the classifiers on two benchmarks in grammatical classification and sentiment analysis. Supplementary materials for this article are available online.