Representation Learning for Natural Language Processing

This open access book provides an overview of the recent advances in representation learning theory, algorithms and applications for natural language processing (NLP). It is divided into three parts. Part I presents the representation learning techniques for multiple language entries, including word...

Full description

Saved in:
Bibliographic Details
Main Authors Liu, Zhiyuan, Lin, Yankai, Sun, Maosong
Format eBook
LanguageEnglish
Published Singapore Springer Nature 2020
Springer Singapore Pte. Limited
Springer
Edition1
Subjects
Artificial intelligence
Big Data
Computational and corpus linguistics
Computational Linguistics
Computer science
Computer Technology
Computing and Information Technology
Data mining
Data Mining and Knowledge Discovery
Deep Learning
Document Representation
Expert systems / knowledge-based systems
Knowledge Representation
Language and Linguistics
Language Arts
Linguistics
Machine Learning
Natural language & machine translation
Natural language and machine translation
Natural Language Processing
Natural language processing (Computer science)
Natural Language Processing (NLP)
Nonfiction
Open Access
Representation Learning
Word Representation
Online AccessGet full text

Cover

Table of Contents:
  • 4.5.2 Relation Extraction -- 4.6 Summary -- References -- 5 RETRACTED CHAPTER: Document Representation -- 6 Sememe Knowledge Representation -- 6.1 Introduction -- 6.1.1 Linguistic Knowledge Graphs -- 6.2 Sememe Knowledge Representation -- 6.2.1 Simple Sememe Aggregation Model -- 6.2.2 Sememe Attention over Context Model -- 6.2.3 Sememe Attention over Target Model -- 6.3 Applications -- 6.3.1 Sememe-Guided Word Representation -- 6.3.2 Sememe-Guided Semantic Compositionality Modeling -- 6.3.3 Sememe-Guided Language Modeling -- 6.3.4 Sememe Prediction -- 6.3.5 Other Sememe-Guided Applications -- 6.4 Summary -- References -- 7 World Knowledge Representation -- 7.1 Introduction -- 7.1.1 World Knowledge Graphs -- 7.2 Knowledge Graph Representation -- 7.2.1 Notations -- 7.2.2 TransE -- 7.2.3 Extensions of TransE -- 7.2.4 Other Models -- 7.3 Multisource Knowledge Graph Representation -- 7.3.1 Knowledge Graph Representation with Texts -- 7.3.2 Knowledge Graph Representation with Types -- 7.3.3 Knowledge Graph Representation with Images -- 7.3.4 Knowledge Graph Representation with Logic Rules -- 7.4 Applications -- 7.4.1 Knowledge Graph Completion -- 7.4.2 Knowledge-Guided Entity Typing -- 7.4.3 Knowledge-Guided Information Retrieval -- 7.4.4 Knowledge-Guided Language Models -- 7.4.5 Other Knowledge-Guided Applications -- 7.5 Summary -- References -- 8 Network Representation -- 8.1 Introduction -- 8.2 Network Representation -- 8.2.1 Spectral Clustering Based Methods -- 8.2.2 DeepWalk -- 8.2.3 Matrix Factorization Based Methods -- 8.2.4 Structural Deep Network Methods -- 8.2.5 Extensions -- 8.2.6 Applications -- 8.3 Graph Neural Networks -- 8.3.1 Motivations -- 8.3.2 Graph Convolutional Networks -- 8.3.3 Graph Attention Networks -- 8.3.4 Graph Recurrent Networks -- 8.3.5 Extensions -- 8.3.6 Applications -- 8.4 Summary -- References
  • Intro -- Preface -- Acknowledgements -- Contents -- Acronyms -- Symbols and Notations -- 1 Representation Learning and NLP -- 1.1 Motivation -- 1.2 Why Representation Learning Is Important for NLP -- 1.3 Basic Ideas of Representation Learning -- 1.4 Development of Representation Learning for NLP -- 1.5 Learning Approaches to Representation Learning for NLP -- 1.6 Applications of Representation Learning for NLP -- 1.7 The Organization of This Book -- References -- 2 Word Representation -- 2.1 Introduction -- 2.2 One-Hot Word Representation -- 2.3 Distributed Word Representation -- 2.3.1 Brown Cluster -- 2.3.2 Latent Semantic Analysis -- 2.3.3 Word2vec -- 2.3.4 GloVe -- 2.4 Contextualized Word Representation -- 2.5 Extensions -- 2.5.1 Word Representation Theories -- 2.5.2 Multi-prototype Word Representation -- 2.5.3 Multisource Word Representation -- 2.5.4 Multilingual Word Representation -- 2.5.5 Task-Specific Word Representation -- 2.5.6 Time-Specific Word Representation -- 2.6 Evaluation -- 2.6.1 Word Similarity/Relatedness -- 2.6.2 Word Analogy -- 2.7 Summary -- References -- 3 Compositional Semantics -- 3.1 Introduction -- 3.2 Semantic Space -- 3.2.1 Vector Space -- 3.2.2 Matrix-Vector Space -- 3.3 Binary Composition -- 3.3.1 Additive Model -- 3.3.2 Multiplicative Model -- 3.4 N-Ary Composition -- 3.4.1 Recurrent Neural Network -- 3.4.2 Recursive Neural Network -- 3.4.3 Convolutional Neural Network -- 3.5 Summary -- References -- 4 Sentence Representation -- 4.1 Introduction -- 4.2 One-Hot Sentence Representation -- 4.3 Probabilistic Language Model -- 4.4 Neural Language Model -- 4.4.1 Feedforward Neural Network Language Model -- 4.4.2 Convolutional Neural Network Language Model -- 4.4.3 Recurrent Neural Network Language Model -- 4.4.4 Transformer Language Model -- 4.4.5 Extensions -- 4.5 Applications -- 4.5.1 Text Classification
  • 9 Cross-Modal Representation -- 9.1 Introduction -- 9.2 Cross-Modal Representation -- 9.2.1 Visual Word2vec -- 9.2.2 Cross-Modal Representation for Zero-Shot Recognition -- 9.2.3 Cross-Modal Representation for Cross-Media Retrieval -- 9.3 Image Captioning -- 9.3.1 Retrieval Models for Image Captioning -- 9.3.2 Generation Models for Image Captioning -- 9.3.3 Neural Models for Image Captioning -- 9.4 Visual Relationship Detection -- 9.4.1 Visual Relationship Detection with Language Priors -- 9.4.2 Visual Translation Embedding Network -- 9.4.3 Scene Graph Generation -- 9.5 Visual Question Answering -- 9.5.1 VQA and VQA Datasets -- 9.5.2 VQA Models -- 9.6 Summary -- References -- 10 Resources -- 10.1 Open-Source Frameworks for Deep Learning -- 10.1.1 Caffe -- 10.1.2 Theano -- 10.1.3 TensorFlow -- 10.1.4 Torch -- 10.1.5 PyTorch -- 10.1.6 Keras -- 10.1.7 MXNet -- 10.2 Open Resources for Word Representation -- 10.2.1 Word2Vec -- 10.2.2 GloVe -- 10.3 Open Resources for Knowledge Graph Representation -- 10.3.1 OpenKE -- 10.3.2 Scikit-Kge -- 10.4 Open Resources for Network Representation -- 10.4.1 OpenNE -- 10.4.2 GEM -- 10.4.3 GraphVite -- 10.4.4 CogDL -- 10.5 Open Resources for Relation Extraction -- 10.5.1 OpenNRE -- References -- 11 Outlook -- 11.1 Introduction -- 11.2 Using More Unsupervised Data -- 11.3 Utilizing Fewer Labeled Data -- 11.4 Employing Deeper Neural Architectures -- 11.5 Improving Model Interpretability -- 11.6 Fusing the Advances from Other Areas -- References -- Correction to: Z. Liu et al., Representation Learning for Natural Language Processing, https://doi.org/10.1007/978-981-15-5573-2
  • 8.2.4 Structural Deep Network Methods -- 8.2.5 Extensions -- 8.2.6 Applications -- 8.3 Graph Neural Networks -- 8.3.1 Motivations -- 8.3.2 Graph Convolutional Networks -- 8.3.3 Graph Attention Networks -- 8.3.4 Graph Recurrent Networks -- 8.3.5 Extensions -- 8.3.6 Applications -- 8.4 Summary -- References -- 9 Cross-Modal Representation -- 9.1 Introduction -- 9.2 Cross-Modal Representation -- 9.2.1 Visual Word2vec -- 9.2.2 Cross-Modal Representation for Zero-Shot Recognition -- 9.2.3 Cross-Modal Representation for Cross-Media Retrieval -- 9.3 Image Captioning -- 9.3.1 Retrieval Models for Image Captioning -- 9.3.2 Generation Models for Image Captioning -- 9.3.3 Neural Models for Image Captioning -- 9.4 Visual Relationship Detection -- 9.4.1 Visual Relationship Detection with Language Priors -- 9.4.2 Visual Translation Embedding Network -- 9.4.3 Scene Graph Generation -- 9.5 Visual Question Answering -- 9.5.1 VQA and VQA Datasets -- 9.5.2 VQA Models -- 9.6 Summary -- References -- 10 Resources -- 10.1 Open-Source Frameworks for Deep Learning -- 10.1.1 Caffe -- 10.1.2 Theano -- 10.1.3 TensorFlow -- 10.1.4 Torch -- 10.1.5 PyTorch -- 10.1.6 Keras -- 10.1.7 MXNet -- 10.2 Open Resources for Word Representation -- 10.2.1 Word2Vec -- 10.2.2 GloVe -- 10.3 Open Resources for Knowledge Graph Representation -- 10.3.1 OpenKE -- 10.3.2 Scikit-Kge -- 10.4 Open Resources for Network Representation -- 10.4.1 OpenNE -- 10.4.2 GEM -- 10.4.3 GraphVite -- 10.4.4 CogDL -- 10.5 Open Resources for Relation Extraction -- 10.5.1 OpenNRE -- References -- 11 Outlook -- 11.1 Introduction -- 11.2 Using More Unsupervised Data -- 11.3 Utilizing Fewer Labeled Data -- 11.4 Employing Deeper Neural Architectures -- 11.5 Improving Model Interpretability -- 11.6 Fusing the Advances from Other Areas -- References
  • 4.5.2 Relation Extraction -- 4.6 Summary -- References -- 5 Document Representation -- 5.1 Introduction -- 5.2 One-Hot Document Representation -- 5.3 Topic Model -- 5.3.1 Latent Dirichlet Allocation -- 5.3.2 Extensions -- 5.4 Distributed Document Representation -- 5.4.1 Paragraph Vector -- 5.4.2 Neural Document Representation -- 5.5 Applications -- 5.5.1 Neural Information Retrieval -- 5.5.2 Question Answering -- 5.6 Summary -- References -- 6 Sememe Knowledge Representation -- 6.1 Introduction -- 6.1.1 Linguistic Knowledge Graphs -- 6.2 Sememe Knowledge Representation -- 6.2.1 Simple Sememe Aggregation Model -- 6.2.2 Sememe Attention over Context Model -- 6.2.3 Sememe Attention over Target Model -- 6.3 Applications -- 6.3.1 Sememe-Guided Word Representation -- 6.3.2 Sememe-Guided Semantic Compositionality Modeling -- 6.3.3 Sememe-Guided Language Modeling -- 6.3.4 Sememe Prediction -- 6.3.5 Other Sememe-Guided Applications -- 6.4 Summary -- References -- 7 World Knowledge Representation -- 7.1 Introduction -- 7.1.1 World Knowledge Graphs -- 7.2 Knowledge Graph Representation -- 7.2.1 Notations -- 7.2.2 TransE -- 7.2.3 Extensions of TransE -- 7.2.4 Other Models -- 7.3 Multisource Knowledge Graph Representation -- 7.3.1 Knowledge Graph Representation with Texts -- 7.3.2 Knowledge Graph Representation with Types -- 7.3.3 Knowledge Graph Representation with Images -- 7.3.4 Knowledge Graph Representation with Logic Rules -- 7.4 Applications -- 7.4.1 Knowledge Graph Completion -- 7.4.2 Knowledge-Guided Entity Typing -- 7.4.3 Knowledge-Guided Information Retrieval -- 7.4.4 Knowledge-Guided Language Models -- 7.4.5 Other Knowledge-Guided Applications -- 7.5 Summary -- References -- 8 Network Representation -- 8.1 Introduction -- 8.2 Network Representation -- 8.2.1 Spectral Clustering Based Methods -- 8.2.2 DeepWalk -- 8.2.3 Matrix Factorization Based Methods