Can AI serve as a substitute for human subjects in software engineering research?

Research within sociotechnical domains, such as software engineering, fundamentally requires the human perspective. Nevertheless, traditional qualitative data collection methods suffer from difficulties in participant recruitment, scaling, and labor intensity. This vision paper proposes a novel appr...

Full description

Saved in:
Bibliographic Details
Published inAutomated software engineering Vol. 31; no. 1; p. 13
Main Authors Gerosa, Marco, Trinkenreich, Bianca, Steinmacher, Igor, Sarma, Anita
Format Journal Article
LanguageEnglish
Published New York Springer US 01.05.2024
Springer Nature B.V
Subjects
Artificial Intelligence
Automation
Collaboration
Computer Science
Data collection
Documentation
Engineering research
Focus groups
Human factors research
Human subjects
Interviews
Language
Large language models
Qualitative analysis
Qualitative research
Researchers
Software engineering
Software Engineering/Programming and Operating Systems
Women
Qualitative research
Foundation models
Large language models
Software engineering
Online AccessGet full text

Cover

More Information
Summary:Research within sociotechnical domains, such as software engineering, fundamentally requires the human perspective. Nevertheless, traditional qualitative data collection methods suffer from difficulties in participant recruitment, scaling, and labor intensity. This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI), especially large language models (LLMs) like ChatGPT and multimodal foundation models. We explore the potential of AI-generated synthetic text as an alternative source of qualitative data, discussing how LLMs can replicate human responses and behaviors in research settings. We discuss AI applications in emulating humans in interviews, focus groups, surveys, observational studies, and user evaluations. We discuss open problems and research opportunities to implement this vision. In the future, an integrated approach where both AI and human-generated data coexist will likely yield the most effective outcomes.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0928-8910
1573-7535
DOI:10.1007/s10515-023-00409-6