Novel Efficient RNN and LSTM-Like Architectures: Recurrent and Gated Broad Learning Systems and Their Applications for Text Classification

• Published in: IEEE transactions on cybernetics Vol. 51; no. 3; pp. 1586 - 1597
• Main Authors: Du, Jie, Vong, Chi-Man, Chen, C. L. Philip
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Broad learning system (BLS); Classification; Computer architecture; Datasets; Evaluation; Information flow; Learning; Learning systems; Logic gates; Recurrent neural networks; sequence information; simultaneous learning; Task analysis; text classification; Training; word importance
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2020.2969705

High accuracy of text classification can be achieved through simultaneous learning of multiple information, such as sequence information and word importance . In this article, a kind of flat neural networks called the broad learning system (BLS) is employed to derive two novel learning methods for text classification, including recurrent BLS (R-BLS) and long short-term memory (LSTM)-like architecture: gated BLS (G-BLS). The proposed two methods possess three advantages: 1) higher accuracy due to the simultaneous learning of multiple information, even compared to deep LSTM that extracts deeper but single information only; 2) significantly faster training time due to the noniterative learning in BLS, compared to LSTM; and 3) easy integration with other discriminant information for further improvement. The proposed methods have been evaluated over 13 real-world datasets from various types of text classification. From the experimental results, the proposed methods achieve higher accuracies than LSTM while taking significantly less training time on most evaluated datasets, especially when the LSTM is in deep architecture. Compared to R-BLS, G-BLS has an extra forget gate to control the flow of information (similar to LSTM) to further improve the accuracy on text classification so that G-BLS is more effective while R-BLS is more efficient.