Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study

• Published in: Journal of software : evolution and process Vol. 37; no. 1
• Main Authors: Wang, Ye, Xue, Weihao, Huang, Qiao, Jiang, Bo, Zhang, Hua
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.01.2025
• Subjects: API recommendation; Application programming interface; Artificial intelligence; BIKER; chain of thought; Chatbots; ChatGPT; Datasets; empirical study; Large language models; MULAREC; Software; Software development
• ISSN: 2047-7473; 2047-7481
• DOI: 10.1002/smr.2765

ABSTRACT As software development grows increasingly complex, application programming interface (API) plays a significant role in enhancing development efficiency and code quality. However, the explosive growth in the number of APIs makes it impossible for developers to become familiar with all of them. In actual development scenarios, developers may spend a significant amount of time searching for suitable APIs, which could severely impact the development process. Recently, the OpenAI's large language model (LLM) based application—ChatGPT has shown exceptional performance across various software development tasks, responding swiftly to instructions and generating high‐quality textual responses, suggesting its potential in API recommendation tasks. Thus, this paper presents an empirical study to investigate the performance of ChatGPT in query‐based API recommendation tasks. Specifically, we utilized the existing benchmark APIBENCH‐Q and the newly constructed dataset as evaluation datasets, selecting the state‐of‐the‐art models BIKER and MULAREC for comparison with ChatGPT. Our research findings demonstrate that ChatGPT outperforms existing approaches in terms of success rate, mean reciprocal rank (MRR), and mean average precision (MAP). Through a manual examination of samples in which ChatGPT exceeds baseline performance and those where it provides incorrect answers, we further substantiate ChatGPT's advantages over the baselines and identify several issues contributing to its suboptimal performance. To address these issues and enhance ChatGPT's recommendation capabilities, we employed two strategies: (1) utilizing a more advanced LLM (GPT‐4) and (2) exploring a new approach—MACAR, which is based on the Chain of Thought methodology. The results indicate that both strategies are effective. This paper presents an empirical study to investigate the performance of ChatGPT in query‐based API recommendation tasks. Our research findings demonstrate that ChatGPT with specific prompt outperforms existing approaches in terms of success rate, mean reciprocal rank (MRR), and mean average precision (MAP). To further improve the performance of ChatGPT, we propose a new approach called MACAR, which utilize multi‐agent collaboration and chain of thought.