RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

• Published in: arXiv.org
• Main Authors: Niu, Xuesong, Shan, Shiguang, Hu, Han, Chen, Xilin
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 04.11.2019
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Head movement; Heart rate; Illumination; Replicating; Representations
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.1910.11515

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HR, available at 'http://vipl.ict.ac.cn/view_database.php?id=15'), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.