Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks

• Published in: ACM transactions on computer-human interaction Vol. 29; no. 2; pp. 1 - 33
• Main Authors: Wang, April Yi, Wang, Dakuo, Drozdal, Jaimie, Muller, Michael, Park, Soya, Weisz, Justin D., Liu, Xuye, Wu, Lingfei, Dugan, Casey
• Format: Journal Article
• Language: English
• Published: 01.04.2022
• ISSN: 1073-0516; 1557-7325
• DOI: 10.1145/3489465

Computational notebooks allow data scientists to express their ideas through a combination of code and documentation. However, data scientists often pay attention only to the code, and neglect creating or updating their documentation during quick iterations. Inspired by human documentation practices learned from 80 highly-voted Kaggle notebooks, we design and implement Themisto, an automated documentation generation system to explore how human-centered AI systems can support human data scientists in the machine learning code documentation scenario. Themisto facilitates the creation of documentation via three approaches: a deep-learning-based approach to generate documentation for source code, a query-based approach to retrieve online API documentation for source code, and a user prompt approach to nudge users to write documentation. We evaluated Themisto in a within-subjects experiment with 24 data science practitioners, and found that automated documentation generation techniques reduced the time for writing documentation, reminded participants to document code they would have ignored, and improved participants’ satisfaction with their computational notebook.