How is the use of technology in education evaluated? A systematic review

• Published in: Computers and education Vol. 133; pp. 27 - 42
• Main Authors: Lai, Jennifer W.M., Bower, Matt
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.05.2019
• Subjects: Applications in subject areas; Evaluation methodologies; Interactive learning environments; Applications in subject areas; Evaluation methodologies; Interactive learning environments
• ISSN: 0360-1315; 1873-782X
• DOI: 10.1016/j.compedu.2019.01.010

There are a large variety of methodologies, contexts and perspectives that have been used to evaluate the use of technology in education. The vast array of literature involving learning technology evaluation makes it challenging to acquire an accurate sense of the different aspects of learning that are evaluated, and the possible approaches that can be used to evaluate them. This study conducted a systematic review of how technology use in education has been evaluated, based on 365 papers published in Computers and Education between 2015 and 2017. The papers analyzed encompassed a diverse range of education levels, disciplines and technologies, that were broadly commensurate with characterizations of the learning technology field from other studies. The analysis found that the evaluation of learning technology use tended to focus on eight themes: learning outcomes, affective elements, behaviors, design, technology elements, pedagogy, presence, and institutional environment. In the majority of studies (66.6%) at least one established instrument was used to evaluate the effectiveness of learning technology usage, however, a wide variety of instruments were observed within sub-themes of evaluation. The 22 instruments used to evaluate an aspect of learning technology usage in more than one study are identified. Evaluation trends for different disciplines, technologies and educational levels are also established. The findings provide an analytical framework that educators and researchers can utilize when evaluating the use of technology in education, and could potentially inform the development of new, more holistic and more robust evaluation methods. •Eight foci of learning technology evaluations emerged from a systematic review.•The main foci were learning outcomes, affective elements, behaviors and design.•Of the 365 papers analyzed, 66.6% used at least one instrument from a prior study.•The 22 instruments used in multiple studies to evaluate sub-categories are reported.•The results assist learning technology evaluation and characterize the field.