An Assessment of ChatGPT on Log Data
Recent development of large language models (LLMs), such as ChatGPT has been widely applied to a wide range of software engineering tasks. Many papers have reported their analysis on the potential advantages and limitations of ChatGPT for writing code, summarization, text generation, etc. However, t...
Saved in:
| Main Authors | , |
|---|---|
| Format | Journal Article |
| Language | English |
| Published |
14.09.2023
|
| Subjects | |
| Online Access | Get full text |
| DOI | 10.48550/arxiv.2309.07938 |
Cover
| Summary: | Recent development of large language models (LLMs), such as ChatGPT has been
widely applied to a wide range of software engineering tasks. Many papers have
reported their analysis on the potential advantages and limitations of ChatGPT
for writing code, summarization, text generation, etc. However, the analysis of
the current state of ChatGPT for log processing has received little attention.
Logs generated by large-scale software systems are complex and hard to
understand. Despite their complexity, they provide crucial information for
subject matter experts to understand the system status and diagnose problems of
the systems. In this paper, we investigate the current capabilities of ChatGPT
to perform several interesting tasks on log data, while also trying to identify
its main shortcomings. Our findings show that the performance of the current
version of ChatGPT for log processing is limited, with a lack of consistency in
responses and scalability issues. We also outline our views on how we perceive
the role of LLMs in the log processing discipline and possible next steps to
improve the current capabilities of ChatGPT and the future LLMs in this area.
We believe our work can contribute to future academic research to address the
identified issues. |
|---|---|
| DOI: | 10.48550/arxiv.2309.07938 |