Semantically Aligned Question and Code Generation for Automated Insight Generation

Automated insight generation is a common tactic for helping knowledge workers, such as data scientists, to quickly understand the potential value of new and unfamiliar data. Unfortunately, automated insights produced by large-language models can generate code that does not correctly correspond (or a...

Full description

Saved in:
Bibliographic Details
Published in2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) pp. 127 - 134
Main Authors Singha, Ananya, Chopra, Bhavya, Khatry, Anirudh, Gulwani, Sumit, Henley, Austin Z., Le, Vu, Parnin, Chris, Singh, Mukul, Verbruggen, Gust
Format Conference Proceeding
LanguageEnglish
Published ACM 20.04.2024
Subjects
alignment
Code-generation
Codes
Conferences
Costs
Filtering
Large language models
LLM
Semantics
Online AccessGet full text
DOI10.1145/3643795.3648381

Cover

More Information
Summary:Automated insight generation is a common tactic for helping knowledge workers, such as data scientists, to quickly understand the potential value of new and unfamiliar data. Unfortunately, automated insights produced by large-language models can generate code that does not correctly correspond (or align) to the insight. In this paper, we leverage the semantic knowledge of large language models to generate targeted and insightful questions about data and the corresponding code to answer those questions. Then through an empirical study on data from Open-WikiTable, we show that embeddings can be effectively used for filtering out semantically unaligned pairs of question and code. Additionally, we found that generating questions and code together yields more diverse questions.
DOI:10.1145/3643795.3648381