Translate this page into:
Contactless and Hassle Free Real Time Heart Rate Measurement with Facial Video
Ajay Vikram Singh, PhD Physical Intelligence Department Max Planck Institute for Intelligent Systems Stuttgart 70569 Germany avsingh@is.mpg.de
This article was originally published by Official Publication of The Simulation Society (TSS), accredited by International Society of Cardiovascular Ultrasound (ISCU) and was migrated to Scientific Scholar after the change of Publisher.
Abstract
In the era of automation, an automatic and noninvasive tool is required for day-to-day health monitoring. Cardiac pulse being one of the vital physiological parameters has drawn attention of researchers for its measurement. In this article, the authors present an approach for automatic cardiac pulse measurement, using a typical webcam. In the presented approach, face region is detected and RGB traces are retrieved for the face region. FFT is applied on the extracted traces to compute power spectrum of the respective traces. The peaks of the power spectrum limited in the band region (0.75–4 Hz) are investigated to provide cardiac pulse measurement. The presented approach is hassle free in nature, and hence, can be adopted even in home environment. Experimental results represent the efficacy of the proposed approach.
Keywords
MedCam
cardiac pulse measurement
health monitoring
Introduction
In the past decade, the usage of user-friendly home health monitoring systems, such as health watches and health wrist bands, etc., has seen eminent increase due to the growing consciousness of people about their health. Talking about health, the recent hike in the catastrophic cardiovascular disease has been a major concern. These cardiovascular measurements highlight not only the presence of heart-related problems but also heart rate variability, which plays a key role in giving information about the physiological state of the person. In this article, the authors propose a noncontact and noninvasive means of estimation of a person's heartbeat in bpm (beats per minute) by observing color variations occurring in the skin during blood flow based on the great work of Poh and colleagues.1 Cardiac pulse measurement helps diagnose heart diseases such as bradycardia and tachycardia,2 and thereby assists in the diagnosis and tracking of medical conditions of individual. It can also be used by athletes who are interested in monitoring their heart rate and improve their performance. Increased heartbeat may be due to physical exercises, illness, and/or mental stress. Until recent developments in technology, the sole methods used for cardiac pulse measurement is via electrocardiogram (ECG) in which users have to strap on bulky sensors (electrodes) on the chest, wrist, etc., or manual measurement by pressing gently with the index and middle fingers at any point on the body where the artery's pulsation is transmitted to the surface. To sum up, attempts so far are nonautomated, difficult to implement in ambient home environment, typically expensive, can increase risk of infection in patients with weak immune system such as neonatal infants, and inappropriate for persons with skin allergies. Using the proposed technique, users can experience remote health monitoring on a regular basis with minimal effort, without visiting a physician, whenever and wherever they want.3 In addition, with the advent of latest technology and growing application of digital image processing, it becomes easier to implement it.
This implementation is based on photoplethysmography (PPG), which is volumetric measurement of an organ by using a pulse oximeter that illuminates the skin and measures changes in absorption of light. Similarly, every time heart beats, the blood flow in the veins increases slightly. Hemoglobin present in blood absorbs light, and so this increase absorbs more light, thereby decreasing the amount of light being reflected by the body (skin). These subtle variations occurring in skin color may not be visible with naked eyes but eventually get embedded in the pixels of image/video recorded by any camera. It is these color variations that may help the doctors calculate the pulse (heartbeat) of the person. Because face is the most expressive and interacting part of the body, the authors used video recording of the face to determine pulses.
Methods
For implementation of this article, the authors recorded the video of the person sitting at a distance of nearly 30 to 50 cm from the laptop. The video was recorded at 15 frames per second in true color (24-bit RGB [red, green, and blue] with three channels) through built-in webcam of the laptop. Because the entire implementation of the project is done in Matlab, the authors also created a user-friendly graphic user interface (GUI) to make the process more interactive. The recorded video of the person is divided into frames, and on each frame, KLT algorithm4 for face detection is applied. The region of interest (ROI) (70% of the face) from the detected face is selected. The ROI is further subdivided into three RGB channels. These channels are the normalized, and then independent component analysis (ICA) was performed on the channels to remove the surrounding noise. On one of the components obtained after ICA, fast Fourier transform (FFT) was applied. Later ideal band pass filtering was applied to confine the frequency in desired range (0.75–4 Hz) and peak detection algorithm to obtain the desired peak frequency. To convert the peak frequency into bpm, it was multiplied by 60.
Theory
Face Detection
Postprocessing and analysis of the video and physiological recordings were done using custom software written in Matlab (The MathWorks, Inc.). An overview of the general steps in this approach to recovering the blood volume pulse is illustrated in the following section. The CV face detection algorithm is based on work by Viola and Jones, as well as Lienhart and Maydt. To track the face over time, this example uses the Kanade-Lucas-Tomasi (KLT) algorithm.4
Kanade-Lucas-Tomasi Algorithm
Kalman-Lucas-Tomasi (KTL) face detection algorithm is used for detection of the face from the first frame of the video and marking certain feature points in the detected face region. Subsequently, these points are reliably tracked in the further video frames, and it results in faster processing of the video and also provides freedom of enough head movements to the user. Alternatively cascade object detector could have been used on every frame, but it is not computationally feasible. It may also fail to detect the face, when the patient turns or tilts the head. This limitation is overcome in case of KLT algorithm (Fig. 1).
- (a) Detected face from the video frame, (b) KLT features used for face tracking during motion.
Region of Interest Extraction
The region of interest (ROI) is selected as a rectangle containing nearly 60% the face region at the first frame of video recording. As shown in Fig. 1, the detected face is captured by webcam. The ROI contains three primary color RGB channels.1
Red, Green, and Blue Trace Separation
The frames obtained from the video are further separated into constituent RGB (red, green, and blue) traces. The ROI is then separated into the three RGB channels and spatially averaged overall pixels in the ROI to yield a red, blue, and green measurement point for each frame and form the raw traces, respectively.
Normalization
The raw RGB traces are normalized as follows:
for each i = 1, 2, 3 where μ and σ are the mean and standard deviation of x t respectively. The normalization results in zero-mean and has unit variance of the raw traces.1
Independent Component Analysis
The traces after normalization are then decomposed into its three independent source signals using independent component analysis (ICA). ICA separates the source signals from linear combination of these signals. In this implementation, the authors used RADICAL (Robust, Accurate, Direct Independent Components Analysis algorithm).5 This method is better than previous ICA algorithms, such as JADE, FastICA, etc. Thus, the RADICAL algorithm is more robust, more accurate, and easier to implement.
Fast Fourier Transform and Bandpass Limit
Finally, the authors applied the fast Fourier transform (FFT) on the selected source signal to obtain the power spectrum. An FFT is a discrete Fourier transform (DFT) but is much faster for calculations. The whole point of using FFT is its high speed in calculating a DFT. The 8-input butterfly diagram of FFT has 12 2-input butterflies and thus 12 × 2 = 24 multiplies. A straight DFT has N × N multiplies or 8 × 8 = 64 multiplies. Thus, there is enough computation efficiency in implementation of FFT. The pulse frequency was designated as the frequency that corresponded to the highest power of the spectrum within an operational frequency band. For these experiments, the authors set the operational range to (0.75–Hz) (corresponding to [45–240 bpm]) to provide a wide range of heart rate measurements.
Graphic User Interface
The graphic user interface (GUI) for the MedCam project was developed in the GUI Layout Editor of the MATLAB 2013a.6 The following components have been used to make the GUI as user friendly as possible (Fig. 2).
- The GUI for MedCam.
The GUI is interactive and interesting. It is provided with various push buttons, static text box, and graphical axes to display the results generated during the implementation of the process. Various push buttons have been used in the GUI to serve different purposes as the name suggests correspondingly for each of the push button (Table 1).
|
S. No. |
Push button |
Function |
|---|---|---|
|
1 |
REC |
To start the video recording |
|
2 |
STOP |
To stop the video recording |
|
3 |
Get Your Pulse Rate |
To start the pulse measurement |
|
4 |
Detected Face |
To display the detected face |
|
5 |
Detected Face Features |
To display the KLT face features |
|
6 |
Red Trace |
To display the red trace |
|
7 |
Green Trace |
To display the green trace |
|
8 |
Blue Trace |
To display the blue trace |
|
9 |
Normalized Red Trace |
To display the normalized red trace |
|
10 |
Normalized Green Trace |
To display the normalized green trace |
|
11 |
Normalized Blue Trace |
To display the normalized blue trace |
|
12 |
Power Spectrum |
To display the power spectrum |
|
13 |
Quit |
To exit the GUI |
Abbreviations: GUI, graphic user interface; KLT, Kanade-Lucas-Tomasi.
Results and Discussion
As shown in the Table 2, the results obtained through MedCam are verified by one of the traditional methods for pulse measurement, that is, by using stethoscope. The readings were taken for three patients: two males and one female under various physical conditions and results were obtained to be 94 to 95% accurate (Fig. 3).
- Flowchart. RGB, red-green-blue; ROI, region of interest.
|
S. No. |
Patient's condition |
MedCam result |
Manual test result |
|---|---|---|---|
|
1 |
Normal (resting) |
72 |
75 |
|
2 |
After swimming |
117 |
100 |
|
3 |
Normal (resting) |
64 |
60 |
|
4 |
After rigorous exercise |
103 |
100 |
|
5 |
Normal |
63 |
60 |
|
6 |
Brisk walking/exercise |
90 |
95 |
Fig. 4 depicts the results obtained after the execution of the MATLAB code. As visible from the above, the importance of normalization process becomes quite evident, because the actual red and green traces are quite distinct from the blue trace, and after the implementation of the normalization function, the three (RGB) traces become quite coherent with each other. Thus, the process of normalization is very effective and important. The power spectrum is obtained after the application of FFT algorithm on the component, which is obtained after implementation of ICA, that is, RADICAL algorithm. The input to the radical algorithm is an array of the normalized RGB channels and the output obtained includes three independent source signal components. It is observed that the second component gives better results due to better signal-to-noise ratio than the other two signals; hence, FFT algorithm is applied on second component. The system can be further extended to send the daily health reports of the participant/patient to his/her health advisor.
- (a) Detected ROI from face, (b) marked feature points (c) power spectrum after FFT and band limiting, (d) red trace, (e) green trace, (f) blue trace, (g) normalized red trace, (h) normalized green trace, (i) normalized blue trace.
Overview of the Process
The detailed overview of the process flow of how RGB traces are extracted for facial ROI, traces generated, and normalized and FFT-based power spectrum generated to predict the accurate cardiac pulse are shown in Fig. 5.
- Process overview.
Conclusion
In this implementation, the authors have demonstrated the semiautomated, advanced and noncontact means of measurement of cardiac pulse measurement, using the typical laptop/computer webcam. This is beneficial for babies or for people with sensitive skin or burn wounds. It will monitor one's health on a regular basis with minimal effort. The previous versions of this implementation were the android/iPhone applications or were intolerant to user's motion. However, this method cannot be used by multiple users at the same time and is also dependent on the lighting conditions; that is, more natural light (sunlight) must be present for more accurate results. Because this technology is the first of its kind (involves noncontact means for heart rate measurement), certain deviation in results can be expected. Further improvements can be made by implementing the code in hardware.
References
- Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express. 2010;18(10):10762-10774.
- [Google Scholar]
- Analysis of heart rate monitoring using a webcam. International Journal of Advanced Research in Computer and Communication Engineering. 2014;3(5):6593-6595.
- [Google Scholar]
- Remote plethysmographic imaging using ambient light. Opt Express. 2008;16(26):21434-21445.
- [Google Scholar]
- Computer Vision System Toolbox. Available at: . Accessed July 26, 2017
- [Publisher]
- The RADICAL ICA Page. Available at: . Accessed July 26, 2017
- [Publisher]
- The Language of Technical Computing. Available at: . Accessed July 26, 2017
- [Publisher]
