How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
ACM/IEEE 46th International Conference on Software Engineering (ICSE 2024) Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitf...
Saved in:
| Main Authors | , , , , |
|---|---|
| Format | Journal Article |
| Language | English |
| Published |
18.12.2023
|
| Subjects | |
| Online Access | Get full text |
| DOI | 10.48550/arxiv.2312.11719 |
Cover
| Abstract | ACM/IEEE 46th International Conference on Software Engineering
(ICSE 2024) Conversational Generative AI (convo-genAI) is revolutionizing Software
Engineering (SE) as engineers and academics embrace this technology in their
work. However, there is a gap in understanding the current potential and
pitfalls of this technology, specifically in supporting students in SE tasks.
In this work, we evaluate through a between-subjects study (N=22) the
effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE
tasks. Our study did not find statistical differences in participants'
productivity or self-efficacy when using ChatGPT as compared to traditional
resources, but we found significantly increased frustration levels. Our study
also revealed 5 distinct faults arising from violations of Human-AI interaction
guidelines, which led to 7 different (negative) consequences on participants. |
|---|---|
| AbstractList | ACM/IEEE 46th International Conference on Software Engineering
(ICSE 2024) Conversational Generative AI (convo-genAI) is revolutionizing Software
Engineering (SE) as engineers and academics embrace this technology in their
work. However, there is a gap in understanding the current potential and
pitfalls of this technology, specifically in supporting students in SE tasks.
In this work, we evaluate through a between-subjects study (N=22) the
effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE
tasks. Our study did not find statistical differences in participants'
productivity or self-efficacy when using ChatGPT as compared to traditional
resources, but we found significantly increased frustration levels. Our study
also revealed 5 distinct faults arising from violations of Human-AI interaction
guidelines, which led to 7 different (negative) consequences on participants. |
| Author | Choudhuri, Rudrajit Steinmacher, Igor Sarma, Anita Liu, Dylan Gerosa, Marco |
| Author_xml | – sequence: 1 givenname: Rudrajit surname: Choudhuri fullname: Choudhuri, Rudrajit – sequence: 2 givenname: Dylan surname: Liu fullname: Liu, Dylan – sequence: 3 givenname: Igor surname: Steinmacher fullname: Steinmacher, Igor – sequence: 4 givenname: Marco surname: Gerosa fullname: Gerosa, Marco – sequence: 5 givenname: Anita surname: Sarma fullname: Sarma, Anita |
| BackLink | https://doi.org/10.48550/arXiv.2312.11719$$DView paper in arXiv |
| BookMark | eNqFjs0KglAUhO-iFv09QKvOC2RdTapVSGgG7RKilVzoqBfyXDma1tun0r7VMMPM8I3FgAyhEHO5tjY7112vFL91bdmOtC0pt3I_EvfQNBAoBo8RbniAKEOIWL_yIitB0aMz6lmCSeCEhKwqXSN4Z9AEF1RMmlK4mqRqVPvgU6oJkdtwKoZJO8TZTydiEfjRMVz2EHHBOlf8iTuYuIdx_je-zpY_-w |
| ContentType | Journal Article |
| Copyright | http://arxiv.org/licenses/nonexclusive-distrib/1.0 |
| Copyright_xml | – notice: http://arxiv.org/licenses/nonexclusive-distrib/1.0 |
| DBID | AKY GOX |
| DOI | 10.48550/arxiv.2312.11719 |
| DatabaseName | arXiv Computer Science arXiv.org |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: GOX name: arXiv.org url: http://arxiv.org/find sourceTypes: Open Access Repository |
| DeliveryMethod | fulltext_linktorsrc |
| ExternalDocumentID | 2312_11719 |
| GroupedDBID | AKY GOX |
| ID | FETCH-arxiv_primary_2312_117193 |
| IEDL.DBID | GOX |
| IngestDate | Tue Jul 22 23:27:12 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-arxiv_primary_2312_117193 |
| OpenAccessLink | https://arxiv.org/abs/2312.11719 |
| ParticipantIDs | arxiv_primary_2312_11719 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-12-18 |
| PublicationDateYYYYMMDD | 2023-12-18 |
| PublicationDate_xml | – month: 12 year: 2023 text: 2023-12-18 day: 18 |
| PublicationDecade | 2020 |
| PublicationYear | 2023 |
| Score | 3.7127328 |
| SecondaryResourceType | preprint |
| Snippet | ACM/IEEE 46th International Conference on Software Engineering
(ICSE 2024) Conversational Generative AI (convo-genAI) is revolutionizing Software
Engineering... |
| SourceID | arxiv |
| SourceType | Open Access Repository |
| SubjectTerms | Computer Science - Human-Computer Interaction Computer Science - Software Engineering |
| Title | How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering |
| URI | https://arxiv.org/abs/2312.11719 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwY2BQSQVWQWlGyaa6wLQB7KCkJSbrJhqZJesCU0-KQUqihZFhKmhzsq-fmUeoiVeEaQQTgwJsL0xiUUVmGeR84KRifWDjwwg0rQg615PZyAjUuXL3j4BMToKP4oKqR6gDtjHBQkiVhJsgAz-0dafgCIkOIQam1DwRhkiP_HIFt8QioHCqQniqvQIwahRCijJBAVmsAOzJgzjAVKCQn6YAOQUaVAQpOHoqZOYpQM8_TVcIBpaX5YlAE5BOEBRlkHdzDXH20AU7Jr4AcnJEPMid8WB3GosxsAD796kSDAomJubAboS5WVIi6PrnlCRgF8YQ2G9MSQEqNLFMMpdkkMBlihRuKWkGLtDN6KCVF4YWMgwsJUWlqbLA-rMkSQ4ciABSoXL_ |
| linkProvider | Cornell University |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Far+Are+We%3F+The+Triumphs+and+Trials+of+Generative+AI+in+Learning+Software+Engineering&rft.au=Choudhuri%2C+Rudrajit&rft.au=Liu%2C+Dylan&rft.au=Steinmacher%2C+Igor&rft.au=Gerosa%2C+Marco&rft.date=2023-12-18&rft_id=info:doi/10.48550%2Farxiv.2312.11719&rft.externalDocID=2312_11719 |