RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation

• Published in: IEEE transactions on image processing Vol. 29; pp. 2409 - 2423
• Main Authors: Niu, Xuesong, Shan, Shiguang, Han, Hu, Chen, Xilin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: end-to-end learning; Estimation; Head; Head movement; Heart rate; Illumination; Image color analysis; Remote heart rate estimation; Replicating; Representations; rPPG; Skin; spatial-temporal representation; Webcams
• DOI: 10.1109/TIP.2019.2947204
• ISSN: 1057-7149; 1941-0042

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HR 1 ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases. 1 VIPL-HR is available at: http://vipl.ict.ac.cn/view_database.php?id=15.