Mutil-Modal Non-Contact Heart Rate Estimation Using Rgb-D Camera

Abstract

Recently, remotely obtaining the photoplethysmogram (PPG) signal to estimate the heart rate of a human, has become a research topic which gains increasing attention. In contrast to the longstanding contact methods (e.g., electrocardiogram (ECG)), the remote PPG methods can tackle the same task with superior convenience and fewer physical constraints. It has been proved in studies that PPG signal will affect the change of color intensity on some body parts such as the face and the wrist. Leveraging this, we propose a new multi-modal method that uses Intel RealSense RGB-D camera to capture RGB/NIR (Near Infrared) facial videos of human subjects to estimate the heart rate in a short segment of time (5~30 second). By combining a series of image and signal processing techniques, e.g., face detection, facial segmentation, Independent Component Analysis (ICA), filtering, Fast Fourier Transform (FFT), and a proposed automatic component selection algorithm, the heart rate from the human face is estimated. Our method works well with slight head motion. The time length of the required facial video is also greatly reduced (traditionally, 30~60 seconds). By experiments, we achieved a RMSE error of 2.38 beat per minute (bpm) for 10-seconds CIELab+NIR videos. This proved that our method can work well under short segment of input in realistic scenarios. In our future work, shorter video clips (e.g., less than 5 seconds) and tolerance of larger head movements would be achieved so that our application can be well-applied in realistic life.