DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels

• Published in: eLife Vol. 10
• Main Authors: Bohnslav, James P, Wimalasena, Nivanthika K, Clausing, Kelsey J, Dai, Yu Y, Yarmolinsky, David A, Cruz, Tomás, Kashlan, Adam D, Chiappe, M Eugenia, Orefice, Lauren L, Woolf, Clifford J, Harvey, Christopher D
• Format: Journal Article
• Language: English
• Published: England eLife Sciences Publications Ltd 02.09.2021; eLife Sciences Publications, Ltd
• Subjects: Animal behavior; Animals; behavior analysis; Biomedical research; Classification; computer vision; Datasets; deep learning; Drosophila melanogaster; Female; Grooming; Humans; Image Processing, Computer-Assisted; Kinetics; Learning algorithms; Machine learning; Male; Mars; Mice; Mice, Inbred C57BL; Motor Activity; Neural networks; Neural Networks, Computer; Neuroscience; Pattern Recognition, Automated; Reproducibility of Results; Researchers; Social Behavior; Software; Supervised Machine Learning; Tools and Resources; Walking; deep learning; mouse; D. melanogaster; neuroscience; behavior analysis; computer vision
• ISSN: 2050-084X; 2050-084X
• DOI: 10.7554/eLife.63377

Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram’s rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram .