Examination of quantitative methods for analyzing data from concept inventories

• Published in: Physical review. Physics education research Vol. 16; no. 1; pp. 010141 - 10153
• Main Authors: Burkholder, Eric, Walsh, Cole, Holmes, N. G.
• Format: Journal Article
• Language: English
• Published: College Park American Physical Society 01.06.2020
• Subjects: Data Analysis; Data Interpretation; Educational Research; Educational Researchers; Effect Size; Exhibits; Learning; Low Achievement; Multiple Regression Analysis; Opportunities; Physics; Pretesting; Pretests Posttests; Questions; Regression (Statistics); Research Methodology; Science Education; Science Tests; Scores; Social Sciences; Statistical analysis; Student Evaluation; Test Results
• ISSN: 2469-9896; 2469-9896
• DOI: 10.1103/PhysRevPhysEducRes.16.010141

Physics education research (PER) has long used concept inventories to investigate student learning over time and to compare performance across various student subpopulations. PER has traditionally used normalized gain to explore these questions but has begun to use established methods from other fields, including Cohen's "d," multiple linear regression, and linear mixed effects models. The choice of analysis method for examining student learning gains in PER is a current subject of debate. We synthesize this debate here by focusing on the research questions and interpretations that can be drawn using various statistical tools. We focus particularly on how results can be interpreted through an equity lens and what questions can and cannot be answered using different methods. Our results demonstrate the importance of clearly defining a research question and using appropriate tools to answer that question.