A Survey of Domain Knowledge Elicitation in Applied Machine Learning

• Published in: Multimodal technologies and interaction Vol. 5; no. 12; p. 73
• Main Authors: Kerrigan, Daniel, Hullman, Jessica, Bertini, Enrico
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.12.2021
• Subjects: Artificial intelligence; Collaboration; Computer science; Crowdsourcing; Decision making; domain expert; domain knowledge; Domains; elicitation; expert knowledge; Interviews; Knowledge acquisition; Machine learning; Researchers; Scientists; Subject specialists; Taxonomy
• ISSN: 2414-4088; 2414-4088
• DOI: 10.3390/mti5120073

Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields.