Deep Word Association: A Flexible Chinese Word Association Method with Iterative Attention Mechanism

• Published in: Pattern Recognition and Computer Vision pp. 112 - 123
• Main Authors: Huang, Yaoxiong, Xie, Zecheng, Liu, Manfei, Zhang, Shuaitao, Jin, Lianwen
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Attention mechanism; Chinese; DropContext; Recurrent neural network; Word association
• ISBN: 3030033376; 9783030033378
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-03338-5_10

Word association is to predict the subsequent words and phrase, acting as a reminder to accelerate the text-editing process. Existing word association models can only predict the next word inflexibly through a given word vocabulary or a simply back-off N-gram language model. Herein, we propose a deep word association system based on attention mechanism with the following contributions: (1) To the best of our knowledge, this is the first investigation of an attention-based recurrent neural network for word association. In the experiments, we provide a comprehensive study on the attention processes for the word association problem; (2) An novel approach, named DropContext, is proposed to solve the over-fitting problem during attention training procedure; (3) Compared with conventional vocabulary-based methods, our word association system can generate an arbitrary-length string of words that are reasonable; (4) Given information on different hierarchies, the proposed system can flexibly generate associated words accordingly.