An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills' (PIAAC 2012) process data

• Published in: Large-scale assessments in education Vol. 12; no. 1; pp. 6 - 22
• Main Authors: Shin, Jinnie, Wang, Bowen, Pinto Junior, Wallace N., Gierl, Mark J.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2024; Springer Nature B.V; SpringerOpen
• Subjects: Assessment; Attention; Education; Educational Policy and Politics; Engagement; Humanities; International Assessment; Law; Learning analytics; Long-short-term-memory (LSTM); PIAAC; Problem solving; Problem-solving performance; PS-TRE; Statistics for Social Sciences; Testing and Evaluation; Log information; Engagement; Long-short-term-memory (LSTM); PIAAC; PS-TRE; Attention; Problem-solving performance
• ISSN: 2196-0739; 2196-0739
• DOI: 10.1186/s40536-024-00194-y

The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee’s performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an “effort-aware” system. The system considers the information regarding the examinee’s task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students’ problem-solving performance.