Planning In Natural Language Improves LLM Search For Code Generation
While scaling training compute has led to remarkable improvements in large language models (LLMs), scaling inference compute has not yet yielded analogous gains. We hypothesize that a core missing component is a lack of diverse LLM outputs, leading to inefficient search due to models repeatedly samp...
Saved in:
Main Authors | , , , , , , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
05.09.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | While scaling training compute has led to remarkable improvements in large
language models (LLMs), scaling inference compute has not yet yielded analogous
gains. We hypothesize that a core missing component is a lack of diverse LLM
outputs, leading to inefficient search due to models repeatedly sampling highly
similar, yet incorrect generations. We empirically demonstrate that this lack
of diversity can be mitigated by searching over candidate plans for solving a
problem in natural language. Based on this insight, we propose PlanSearch, a
novel search algorithm which shows strong results across HumanEval+, MBPP+, and
LiveCodeBench (a contamination-free benchmark for competitive coding).
PlanSearch generates a diverse set of observations about the problem and then
uses these observations to construct plans for solving the problem. By
searching over plans in natural language rather than directly over code
solutions, PlanSearch explores a significantly more diverse range of potential
solutions compared to baseline search methods. Using PlanSearch on top of
Claude 3.5 Sonnet achieves a state-of-the-art pass@200 of 77.0% on
LiveCodeBench, outperforming both the best score achieved without search
(pass@1 = 41.4%) and using standard repeated sampling (pass@200 = 60.6%).
Finally, we show that, across all models, search algorithms, and benchmarks
analyzed, we can accurately predict performance gains due to search as a direct
function of the diversity over generated ideas. Code can be found at
https://github.com/scaleapi/plansearch. |
---|---|
DOI: | 10.48550/arxiv.2409.03733 |