FlowNL: Asking the Flow Data in Natural Languages

• Published in: IEEE transactions on visualization and computer graphics Vol. 29; no. 1; pp. 1 - 11
• Main Authors: Huang, Jieying, Xi, Yang, Hu, Junnan, Tao, Jun
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Data visualization; declarative grammar; Domains; Empirical analysis; Engines; Flow distribution; Flow visualization; interactive exploration; Interactive systems; Language; Learning; Natural language; Natural language (computers); natural language interface; Natural languages; Programming languages; Questions; Scalars; Semantics; Spirals; Subject specialists; Task analysis; Visual analytics; Visualization
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2022.3209453

Flow visualization is essentially a tool to answer domain experts' questions about flow fields using rendered images. Static flow visualization approaches require domain experts to raise their questions to visualization experts, who develop specific techniques to extract and visualize the flow structures of interest. Interactive visualization approaches allow domain experts to ask the system directly through the visual analytic interface, which provides flexibility to support various tasks. However, in practice, the visual analytic interface may require extra learning effort, which often discourages domain experts and limits its usage in real-world scenarios. In this paper, we propose FlowNL, a novel interactive system with a natural language interface. FlowNL allows users to manipulate the flow visualization system using plain English, which greatly reduces the learning effort. We develop a natural language parser to interpret user intention and translate textual input into a declarative language. We design the declarative language as an intermediate layer between the natural language and the programming language specifically for flow visualization. The declarative language provides selection and composition rules to derive relatively complicated flow structures from primitive objects that encode various kinds of information about scalar fields, flow patterns, regions of interest, connectivities, etc. We demonstrate the effectiveness of FlowNL using multiple usage scenarios and an empirical evaluation.