Skeleton-based image feature extraction for automated behavioral analysis in human-animal relationship tests

• Published in: Applied animal behaviour science Vol. 277; p. 106347
• Main Authors: Oczak, Maciej, Rault, Jean-Loup, Truong, Suzanne, Schmitt, Oceane
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.08.2024
• Subjects: animal behavior; automation; Computer vision; Distance; females; head; Human-animal; human-animal relations; humans; Key point detection; Object detection; people; skeleton; swine; Computer vision; Human-animal; Key point detection; Distance; Object detection
• ISSN: 0168-1591; 1872-9045
• DOI: 10.1016/j.applanim.2024.106347

Arena tests are used to address various research questions related to animal behavior and human-animal relationships; e.g. how animals perceive specific human beings or people in general. Recent advancements in computer vision, specifically in application of key point detection models, might offer a possibility to extract variables that are the most often recorded in these tests in an automated way. The objective of this study was to measure two variables in human-pig arena test with computer vision techniques, i.e. distance between the subjects and pig’s visual attention proxy towards pen areas including a human. Human-pig interaction tests were organized inside a test arena measuring 147 × 168 cm. Thirty female pigs took part in the arena tests from 8 to 11 weeks of age, for a total of 210 tests (7 tests per pig), each with a 10-min duration. In total, 35 hours of human-pig interaction tests were video-recorded. To automatically detect human and pig skeletons, 4 models were trained on 100 images of labeled data, i.e. two YOLOv8 models to detect human and pig locations and two VitPose models to detect their skeletons. Models were validated on 50 images. The best performing models were selected to extract human and pig skeletons on recorded videos. Human-pig distance was calculated as the shortest Euclidean distance between all key points of the human and the pig. Visual attention proxy towards selected areas of the arena were calculated by extracting the pig’s head direction and calculating the intersection of a line indicating the heads direction and lines specifying the areas i.e. either lines of the quadrangles for the entrance and the window or lines joining the key points of the human skeleton. The performance of the YOLOv8 for detection of the human and the pig was 0.86 mAP and 0.85 mAP, respectively, and for the VitPose model 0.65 mAP and 0.78 mAP, respectively. The average distance between the human and the pig was 31.03 cm (SD = 35.99). Out of the three predefined areas in the arena, pigs spend most of their time with their head directed toward the human, i.e. 12 hrs 11 min (34.83 % of test duration). The developed method could be applied in human-animal relationship tests to automatically measure the distance between a human and a pig or another animal, visual attention proxy or other variables of interest. •We automatised skeleton-based image feature extraction in an arena test.•The human’s and the pig’s skeletons were detected with 0.65 mAP and 0.78 mAP.•The average distance between the human and the pig was 31.03 cm.•Pigs spent 34.83 % of the test duration with their head directed toward the human.•The developed method can reduce the need for time-consuming manual observations.