ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to...
Saved in:
Main Authors | , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
13.03.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Charts provide visual representations of data and are widely used for
analyzing information, addressing queries, and conveying insights to others.
Various chart-related downstream tasks have emerged recently, such as
question-answering and summarization. A common strategy to solve these tasks is
to fine-tune various models originally trained on vision tasks language.
However, such task-specific models are not capable of solving a wide range of
chart-related tasks, constraining their real-world applicability. To overcome
these challenges, we introduce ChartInstruct: a novel chart-specific
vision-language Instruction-following dataset comprising 191K instructions
generated with 71K charts. We then present two distinct systems for instruction
tuning on such datasets: (1) an end-to-end model that connects a vision encoder
for chart understanding with a LLM; and (2) a pipeline model that employs a
two-step approach to extract chart data tables and input them into the LLM. In
experiments on four downstream tasks, we first show the effectiveness of our
model--achieving a new set of state-of-the-art results. Further evaluation
shows that our instruction-tuning approach supports a wide array of real-world
chart comprehension and reasoning scenarios, thereby expanding the scope and
applicability of our models to new kinds of tasks. |
---|---|
DOI: | 10.48550/arxiv.2403.09028 |