Behavior-Based Video Summarization System for Dog Health and Welfare Monitoring

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 6; p. 2892
• Main Authors: Atif, Othmane, Lee, Jonguk, Park, Daihee, Chung, Yongwha
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 01.03.2023; MDPI
• Subjects: Animal behavior; Animals; Background noise; computer vision; Data Collection; Discriminant analysis; dog behavior recognition; dog health and welfare; Dogs; Households; Mental health; Methods; Motion perception; Neural networks; Noise reduction; Optical flow (image analysis); Pets; Video data; video monitoring system; video summarization; Visualization; South Korea; visualization; dog behavior recognition; video monitoring system; dog health and welfare; video summarization; computer vision
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23062892

The popularity of dogs has been increasing owing to factors such as the physical and mental health benefits associated with raising them. While owners care about their dogs' health and welfare, it is difficult for them to assess these, and frequent veterinary checkups represent a growing financial burden. In this study, we propose a behavior-based video summarization and visualization system for monitoring a dog's behavioral patterns to help assess its health and welfare. The system proceeds in four modules: (1) a video data collection and preprocessing module; (2) an object detection-based module for retrieving image sequences where the dog is alone and cropping them to reduce background noise; (3) a dog behavior recognition module using two-stream EfficientNetV2 to extract appearance and motion features from the cropped images and their respective optical flow, followed by a long short-term memory (LSTM) model to recognize the dog's behaviors; and (4) a summarization and visualization module to provide effective visual summaries of the dog's location and behavior information to help assess and understand its health and welfare. The experimental results show that the system achieved an average F1 score of 0.955 for behavior recognition, with an execution time allowing real-time processing, while the summarization and visualization results demonstrate how the system can help owners assess and understand their dog's health and welfare.