Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

• Published in: European journal of engineering education Vol. 44; no. 1-2; pp. 222 - 233
• Main Authors: Goncher, Andrea M., Boles, Wageeh
• Format: Journal Article
• Language: English
• Published: Abingdon Taylor & Francis 04.03.2019; Taylor & Francis Ltd
• Subjects: Comparative Analysis; Concept Formation; Concept inventories; conceptual understanding; Educational evaluation; Electrical engineering; Electrical occupations; Engineering Education; Evaluation Methods; Feedback (Response); Foreign Countries; Fundamental Concepts; Guessing (Tests); Guidelines; Higher education; Language Role; Misconceptions; Multiple choice; Multiple Choice Tests; Scores; Student assessment; Student Evaluation; Students; Text analysis; Undergraduate Students; Vector analysis; Writing (Composition); Australia
• ISSN: 0304-3797; 1469-5898
• DOI: 10.1080/03043797.2017.1410523

Concept inventories (CIs) are assessment instruments designed to measure students' conceptual understanding of fundamental concepts in particular fields. CIs utilise multiple-choice questions (MCQs), and specifically designed response selections, to help identify misconceptions. One shortcoming of this assessment instrument is that it fails to provide evidence of the causes of the misconceptions, or the nature of students' conceptual understanding. In this article, we present the results of conducting textual analysis on students' written explanations in order to provide better judgements into their conceptual understanding. We compared students' MCQ scores in Signals and Systems Concept Inventory questions, with the textual analysis utilising vector analysis approaches. Our analysis of the textual data provided the ability to detect answers that students identified as a 'guessed' response. However, the analysis was unable to detect if conceptually correct ideas existed within the 'guessed' responses. The presented approach can be used as a framework to analyse assessment instruments that utilise textual, short-answer responses. This analysis framework is best suited for the restricted conditions imposed by the short-answer structure.