Compositional Sentence Representation from Character Within Large Context Text

• Published in: Neural Information Processing Vol. 10635; pp. 674 - 685
• Main Authors: Kim, Geonmin, Lee, Hwaran, Kim, Bokyeong, Lee, Soo-young
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2017; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Dialogue act; Hierarchical recurrent neural network; Hierarchy-wise learning; Inter-sentence dependency; Rare word
• ISBN: 3319700952; 9783319700953
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-70096-0_69

This paper describes a Hierarchical Composition Recurrent Network (HCRN) consisting of a 3-level hierarchy of compositional models: character, word and sentence. This model is designed to overcome two problems of representing a sentence on the basis of a constituent word sequence. The first is a data sparsity problem when estimating the embedding of rare words, and the other is no usage of inter-sentence dependency. In the HCRN, word representations are built from characters, thus resolving the data-sparsity problem, and inter-sentence dependency is embedded into sentence representation at the level of sentence composition. We propose a hierarchy-wise language learning scheme in order to alleviate the optimization difficulties when training deep hierarchical recurrent networks in an end-to-end fashion. The HCRN was quantitatively and qualitatively evaluated on a dialogue act classification task. In the end, the HCRN achieved the state-of-the-art performance with a test error rate of 22.7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\%$$\end{document} for dialogue act classification on the SWBD-DAMSL database.