The human touch: How non-expert users perceive, interpret, and fix topic models

• Published in: International journal of human-computer studies Vol. 105; pp. 28 - 42
• Main Authors: Lee, Tak Yeon, Smith, Alison, Seppi, Kevin, Elmqvist, Niklas, Boyd-Graber, Jordan, Findlater, Leah
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2017
• Subjects: Mixed-initiative interaction; Topic modeling; User study; Topic modeling; Mixed-initiative interaction; User study
• ISSN: 1071-5819; 1095-9300
• DOI: 10.1016/j.ijhcs.2017.03.007

Topic modeling is a common tool for understanding large bodies of text, but is typically provided as a “take it or leave it” proposition. Incorporating human knowledge in unsupervised learning is a promising approach to create high-quality topic models. Existing interactive systems and modeling algorithms support a wide range of refinement operations to express feedback. However, these systems’ interactions are primarily driven by algorithmic convenience, ignoring users who may lack expertise in topic modeling. To better understand how non-expert users understand, assess, and refine topics, we conducted two user studies—an in-person interview study and an online crowdsourced study. These studies demonstrate a disconnect between what non-expert users want and the complex, low-level operations that current interactive systems support. In particular, our findings include: (1) analysis of how non-expert users perceive topic models; (2) characterization of primary refinement operations expected by non-expert users and ordered by relative preference; (3) further evidence of the benefits of supporting users in directly refining a topic model; (4) design implications for future human-in-the-loop topic modeling interfaces. •User studies show a disconnect between what non-expert users want and what human-in-the-loop topic modeling systems support.•We identify a set of important refinement operations that should be included to best support non-expert users.•Findings highlight patterns in how non-experts interpret topics and apply refinement operations to individual topics.•Users perceive their refinements to improve topic model quality; computed topic coherence aligns with this perception.•These findings should guide efforts in algorithmic work on human-in-the-loop topic modeling.