Automating creativity assessment with SemDis: An open platform for computing semantic distance

• Published in: Behavior research methods Vol. 53; no. 2; pp. 757 - 780
• Main Authors: Beaty, Roger E., Johnson, Dan R.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2021; Springer Nature B.V
• Subjects: Automation; Behavioral Science and Psychology; Cognitive Psychology; Computer applications; Creativity; Humans; Information processing; Psychology; Psychometrics; Reproducibility of Results; Semantics; Semantic distance; Divergent thinking; Creativity; Word association; Assessment
• ISSN: 1554-3528; 1554-351X; 1554-3528
• DOI: 10.3758/s13428-020-01453-w

Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants’ responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitations—labor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)—raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Studies 1–3) and novelty/creativity ratings from two word association tasks (Studies 4–5). We find that a latent semantic distance factor—comprised of the common variance from five semantic models—reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation ( https://osf.io/gz4fc/ ).