Towards automatic annotation of collaborative problem‐solving skills in technology‐enhanced environments

• Published in: Journal of computer assisted learning Vol. 38; no. 5; pp. 1434 - 1447
• Main Authors: Flor, Michael, Andrews‐Todd, Jessica
• Format: Journal Article
• Language: English
• Published: Chichester, UK John Wiley & Sons, Inc 01.10.2022; Wiley; Wiley Subscription Services, Inc
• Subjects: 21st century; Accuracy; Annotations; Artificial Intelligence; assessment; Assessments; automated classification; Automation; Classification; Classifiers; Collaboration; collaborative problem solving; Computational Linguistics; Cooperative Learning; discourse; Documentation; Educational Assessment; Electronic Learning; Electronics; Engineering education; Learning Activities; Machine learning; Man Machine Systems; Messages; ontology; Problem Solving; Semantics; Skills; Student Participation; Teams; Teamwork; Technology Uses in Education
• ISSN: 0266-4909; 1365-2729
• DOI: 10.1111/jcal.12689

Background Collaborative problem solving (CPS) is important for success in the 21st century, especially for teamwork and communication in technology‐enhanced environments. Measurement of CPS skills has emerged as an essential aspect in educational assessment. Modern research in CPS relies on theory‐driven measurements that are usually carried out as manual annotations over recorded logs of collaborative activities. However, manual annotation has limited scalability and is not conductive towards CPS assessments at scale. Objective We explore possibilities for automated annotation of actions in collaborative‐teams, especially chat messages. We evaluate two approaches that employ machine learning for automated classification of CPS events. Method Data were collected from engineering, physics and electronics students' participation in a simulation‐based task on electronics concepts, in which participants communicated via text‐chat messages. All task activities were logged and time stamped. Data have been manually classified for the CPS skills, using an ontology that includes both social and cognitive dimensions. In this article, we describe computational linguistic methods for automatically classifying the CPS skills from logged data, with a view towards automating CPS assessments. Results We applied two machine learning methods to our data. A Naïve Bayes classifier has been previously used in CPS research, but it is only moderately successful on our data. We also present a k‐nearest‐neighbours (kNN) classifier that uses distributional semantic models for measuring text similarity. This classifier shows strong agreement between automated and human annotations. The study also demonstrates that automatic spelling correction and slang normalization of chat texts are useful for accurate automated annotation. Implications Our results suggest that a kNN classifier can be very effective for accurate annotation of CPS events. It achieves reasonably strong results even when trained on only half of the available data. This shows a promise towards reduction of manual data annotation for CPS measurement. Lay Description What is already known about this topic Collaborative problem solving (CPS) is important for success in the 21st century, especially when teamwork and communication occur in technology‐enhanced environments. Measurement of CPS skills requires classification and annotation of team interactions, in particular their communications, such as chat messages. What this paper adds We explore possibilities for automated classification of CPS activities using machine learning approaches. Implications for practice One of the approaches we explored is particularly effective and can help reduce the amount of manual annotations needed for CPS measurements.