Using Natural Language Processing Tools to Classify Student Responses to Open-Ended Engineering Problems in Large Classes

• Published in: Association for Engineering Education - Engineering Library Division Papers p. 24.1338.1
• Main Author: Verleger, Matthew A
• Format: Conference Proceeding
• Language: English
• Published: Atlanta American Society for Engineering Education-ASEE 15.06.2014
• Subjects: Algorithms; Classification; Decision trees; Engineering education; Natural language processing

Using Natural Language Processing Tools to Classify Student Responses to Open-Ended Engineering Problems in Large ClassesPeer review can be a beneficial pedagogical tool for providing students both feedback and variedperspectives. Despite being a valuable tool, the best mechanism for assigning reviewers toreviewees is still often blind random assignment. This research represents the first step in alarger effort to find an improved method for matching reviewers to reviewees. By automatingthe classification of student work, reviewer quality and reviewee need can be assessed. With thatassessment, the best reviewers can be assigned to the neediest teams, while the most self-sufficient teams can be assigned reviewers who may need to see higher quality work.The purpose of this paper is to present the preliminary findings from an effort to classify studentteam performance on Model-Eliciting Activities (MEAs) using natural language processingtools. MEAs are realistic, open-ended, client-driven engineering problems where teams ofstudents produce a written document describing the steps of how to solve the problem.Archival data containing expert evaluations to MEAs were used to test different natural languageprocessing tools in an attempt to identify which tools could most accurately assign scores similarto an expert. The research did not re-implement the selected algorithms, but rather used off-the-shelf libraries to explore the value of their application to this context.Using a split-sample training-testing set, the “Bagged Decision Tree” and “Random Forest”algorithms were used to classify sample solutions against 11 MEA rubric dimensions.Performance on each rubric item averaged between 60% and 85% accurate, depending on theitem. The implementation of these algorithms also revealed words and phrases commonly usedin higher quality samples.This paper will focus on how the data was obtained and prepared, how the different algorithmswere utilized, how the algorithms performed in the classification tests, what the results indicateabout our implementation of MEAs and how the results will be informing the next stages of theresearch project.