Estimation of individuals’ collaborative problem solving ability in computer-based assessment

• Published in: Education and information technologies Vol. 29; no. 1; pp. 483 - 515
• Main Authors: Li, Meijuan, Liu, Hongyun, Cai, Mengfei, Yuan, Jianlin
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.01.2024; Springer; Springer Nature B.V
• Subjects: Analysis; Collaboration; Computer Appl. in Social and Behavioral Sciences; Computer Assisted Testing; Computer Science; Computers and Education; Education; Educational Environment; Educational Technology; Grade 8; Information Systems Applications (incl.Internet); Item Response Theory; Problem solving; Social networks; User Interfaces and Human Computer Interaction; China; Individuals’ Collaborative problem-solving ability; Human-to-human Collaborative Problem Solving; a mean correction method; Computer-based Assessment
• ISSN: 1360-2357; 1573-7608
• DOI: 10.1007/s10639-023-12271-w

In the human-to-human Collaborative Problem Solving (CPS) test, students’ problem-solving process reflects the interdependency among partners. The high interdependency in CPS makes it very sensitive to group composition. For example, the group outcome might be driven by a highly competent group member, so it does not reflect all the individual performances, especially for a low-ability member. As a result, how to effectively assess individuals’ performances has become a challenging issue in educational measurement. This research aims to construct the measurement model to estimate an individual’s collaborative problem-solving ability and correct the impact of partners’ abilities. First, 175 eighth graders’ dyads were divided into six cooperative groups with different levels of problem-solving (PS) ability combinations (i.e., high-high, high-medium, high-low, medium-medium, medium–low, and low-low). Then, they participated in the test of three CPS tasks, and the log data of the dyads were recorded. We applied Multidimensional Item Response Theory (MIRT) measurement models to estimate an individual’s CPS ability and proposed a mean correction method to correct the impact of group composition on individual ability. Results show that (1) the multidimensional IRT model fits the data better than the multidimensional IRT model with the testlet effect; (2) the mean correction method significantly reduced the impact of group composition on obtained individual ability. This study not only successfully increased the validity of individuals’ CPS ability measurement but also provided useful guidelines in educational settings to enhance individuals’ CPS ability and promote an individualized learning environment.