Scene labeling with LSTM recurrent neural networks

• Published in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3547 - 3555
• Main Authors: Wonmin Byeon, Breuel, Thomas M., Raue, Federico, Liwicki, Marcus
• Format: Conference Proceeding Journal Article
• Language: English
• Published: IEEE 01.06.2015
• Subjects: Accuracy; Classification; Feedforward neural networks; Networks; Recurrent neural networks; Roads; Segmentation; State of the art; Texture; Two dimensional; Weaving
• ISSN: 1063-6919; 1063-6919; 2575-7075
• DOI: 10.1109/CVPR.2015.7298977

This paper addresses the problem of pixel-level segmentation and classification of scene images with an entirely learning-based approach using Long Short Term Memory (LSTM) recurrent neural networks, which are commonly used for sequence classification. We investigate two-dimensional (2D) LSTM networks for natural scene images taking into account the complex spatial dependencies of labels. Prior methods generally have required separate classification and image segmentation stages and/or pre- and post-processing. In our approach, classification, segmentation, and context integration are all carried out by 2D LSTM networks, allowing texture and spatial model parameters to be learned within a single model. The networks efficiently capture local and global contextual information over raw RGB values and adapt well for complex scene images. Our approach, which has a much lower computational complexity than prior methods, achieved state-of-the-art performance over the Stanford Background and the SIFT Flow datasets. In fact, if no pre- or post-processing is applied, LSTM networks outperform other state-of-the-art approaches. Hence, only with a single-core Central Processing Unit (CPU), the running time of our approach is equivalent or better than the compared state-of-the-art approaches which use a Graphics Processing Unit (GPU). Finally, our networks' ability to visualize feature maps from each layer supports the hypothesis that LSTM networks are overall suited for image processing tasks.