Reverse-engineering bar charts using neural networks

• Published in: Journal of visualization Vol. 24; no. 2; pp. 419 - 435
• Main Authors: Zhou, Fangfang, Zhao, Yong, Chen, Wenjiang, Tan, Yijing, Xu, Yaqi, Chen, Yi, Liu, Chao, Zhao, Ying
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.04.2021; Springer Nature B.V
• Subjects: Charts; Classical and Continuum Physics; Coders; Computer Imaging; Encoders-Decoders; Engineering; Engineering Fluid Dynamics; Engineering Thermodynamics; Graphs; Heat and Mass Transfer; Information retrieval; Neural networks; Object recognition; Pattern Recognition and Graphics; Recurrent neural networks; Regular Paper; Robustness (mathematics); Vision; Information extraction; Bar chart; Neural network; Reverse engineering
• ISSN: 1343-8875; 1875-8975
• DOI: 10.1007/s12650-020-00702-6

Reverse-engineering bar charts extract textual and numeric information from the visual representations of bar charts to support application scenarios that require the underlying information. In this paper, we propose a neural network-based method for reverse-engineering bar charts. We adopt a neural network-based object detection model to simultaneously localize and classify textual information. This approach improves the efficiency of textual information extraction. We design an encoder-decoder framework that integrates convolutional and recurrent neural networks to extract numeric information. We further introduce an attention mechanism into the framework to achieve high accuracy and robustness. Synthetic and real-world datasets are used to evaluate the effectiveness of the method. To the best of our knowledge, this work takes the lead in constructing a complete neural network-based method of reverse-engineering bar charts. Graphic abstract