Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages

• Main Authors: Zhang, William, Leon, Maria, Xu, Ryan, Cardenas, Adrian, Wissink, Amelia, Martin, Hanna, Srikanth, Maya, Dorogi, Kaya, Valadez, Christian, Perez, Pedro, Grijalva, Citlalli, Zhang, Corey, Santolucito, Mark
• Format: Journal Article
• Language: English
• Published: 01.09.2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Programming Languages; Computer Science - Software Engineering
• DOI: 10.48550/arxiv.2409.00856

Node-based programming languages are increasingly popular in media arts coding domains. These languages are designed to be accessible to users with limited coding experience, allowing them to achieve creative output without an extensive programming background. Using LLM-based code generation to further lower the barrier to creative output is an exciting opportunity. However, the best strategy for code generation for visual node-based programming languages is still an open question. In particular, such languages have multiple levels of representation in text, each of which may be used for code generation. In this work, we explore the performance of LLM code generation in audio programming tasks in visual programming languages at multiple levels of representation. We explore code generation through metaprogramming code representations for these languages (i.e., coding the language using a different high-level text-based programming language), as well as through direct node generation with JSON. We evaluate code generated in this way for two visual languages for audio programming on a benchmark set of coding problems. We measure both correctness and complexity of the generated code. We find that metaprogramming results in more semantically correct generated code, given that the code is well-formed (i.e., is syntactically correct and runs). We also find that prompting for richer metaprogramming using randomness and loops led to more complex code.