Heart Rate Variability using the phone's camera

No doubt the main inconvenience of HRV-based applications is the need for a heart rate monitor. Even when you have one, performing your daily measurement can be a burden (sensor needs to be wet and comfort is clearly an issue). 

Can smartphones come to the rescue? Current generation phones include both a camera and a light emitting diode, which can be used for reflection based bio-optical imaging. 

The technique is called photoplethysmography (PPG for short) and consists in detecting changes in blood volume during a cardiac cycle, by illuminating the skin and measuring changes in light absorption. PPG has become quite a popular non-invasive method for extracting physiological measurements such as heart rate and oxygen saturation. However, most applications today focus simply on heart rate, and it is not clear from literature if HRV features can also be reliably extracted using a phone's camera [1]. 

Here is the good news: it is indeed possible to achieve good accuracy in HRV measurements using this technique, but the methods needed are slightly more complicated than acquiring a video and computing peak detection on the PPG signal (which is sufficient for heart rate measurement). This post covers the steps involved in the implementation of Camera HRV, the iPhone app I developed to measure HRV using the phone's camera. The algorithms are part of HRV4Training since version 3.2.

Later in 2017, we have also validated these methods and published the results with respect to ECG and Polar straps, showing equivalency. You can find the full paper here.

Overview

1 - Data acquisition from the phone's camera
2 - Filtering & smoothing
3 - Resampling with cubic spline interpolation
4 - Peak detection
5 - Artifact removal and features extraction
6 - Comparison with heart rate monitors (Polar H7)
7 - Tips

Signal processing pipeline:

All algorithms were first tested in MATLAB and then implemented in iOS.

1 - Data acquisition from the phone's camera

The iPhone's camera can be configured to acquire videos at 30 frames per second, which means a 30 Hz signal. This is the first issue to face, since accurate HRV measurements require sampling frequencies around 200 Hz [2]. Camera HRV gets the RGB values  from the camera frame, averages them over the whole frame, and finally converts RGB to HSV values. HSV values are often used in image processing since they better separate the intensity from the color information. In our case, we are really interested only in intensity, so we will use the hue value of the HSV color space:

2 - Filtering & smoothing

The acquired PPG signal needs some further processing. More specifically, I applied a fourth order Butterworth band pass filter. The band pass filter helps in both removing the DC component, due to finger movement or changes in venous pressure, and also high frequency noise. The result of the filtering is shown in the next plot: 

The final plot includes a five samples moving average to further smooth the signal and ease peak detection.

3 - Resampling with cubic spline interpolation

As mentioned before, one of the biggest issues for HRV analysis is the low resolution of the signal, due to the limited frame rate we can obtain from the phone's camera (i.e. 30 Hz). For Camera HRV I implemented cubic spline interpolation at 180 Hz, which is fast enough for the iPhone to compute, and significantly improves the resolution of the signal. The following plot shows the resolution issue and the improved signal after cubic spline interpolation:

4 - Peak detection

Peak detection is performed using a simple slope inversion algorithm, plus a local peak search to avoid detecting double peaks due to the nature of the PPG signal:

5 - Artifact removal and features extraction

Peak to peak intervals are corrected for artifacts, before features extraction. Intervals correction prevents artifacts due to ectopic beats or motion from affecting features computation, as often reported in literature for HRV analysis. Especially for the PPG signal, intervals correction is very important since the signal is very sensitive to motion artifacts, and HRV features risk to get completely unreliable even when heart rate is quite stable.

Interval correction is performed by analyzing intervals in windows of 10 seconds, and discarding 1) all intervals differing more than 20% from the median of the intervals over the current window, 2) all intervals belonging to a window where we have less than 4 intervals detected or more than 50% intervals removed (i.e. too much noise for a reliable signal). 

After interval correction, features extraction is performed in standard ways, as described in a previous post.

6 - Comparisons with heart rate monitors

Comparisons are against a commercial heart rate monitor, Polar's H7. I used polar's H7 because I already validated it's accuracy with respect to a device able to capture the full ECG (see this post, hardware section), and it also allowed me to directly compare the output with the one of Camera HRV, by developing an application able to record both at the same time.

Heart rate during rest and paced breathing
The following plot shows heart rate recorded by the H7 and Camera HRV. This is the instantaneous HR received every second by BLE HR monitors, however it is not clear if the value is derived only from the last RR-interval or if it is an average of the last few seconds. In Camera HRV I average the last three seconds of data and update the information every second. This might explain small differences between the two. However, both during rest and paced breathing (second part of the plot, showing much greater variations in HR), the correlation between the two signals is high:

Correlation coefficient: 0.97

The following video shows both the PPG signal and the instantaneous HR as received from the Polar H7 (left) and computed by Camera HRV (right). The recording is not the same shown above in the heart rate plot.

Heart rate post exercise
The following plot shows 2 minutes and a half of heart rate post exercise (25 minutes of biking (average heart rate ~130 bpm)). As expected, heart rate can be acquired with high accuracy, even at higher intensities:

Correlation coefficient: 0.92

RR-intervals
In the next plot we can see RR-intervals from the rest, paced breathing and recovery sequences (they did not happen in this order, I just put together all the data in a single time series). Peak to peak intervals are plotted without and with artifact correction. The third plot highlights some successful corrections (interval correction is always enabled in the commercial version of Camera HRV):

Features comparison
Features extraction is the last step to confirm the validity of Camera HRV as a tool able to assess not only heart rate but also heart rate variability. Features were extracted over 30 seconds windows, using the rest, paced breathing and recovery data. The following plots show the relation between HRV features (AVNN, SDNN, rMSSD and pNN50) computed using Polar's H7 and Camera HRV. Features computed using Camera HRV include RR-interval correction. All plots include a time series of the features and a comparison of the values under different conditions.

These plots show that Camera HRV is able to measure heart rate variability under different conditions, capturing small changes in HRV and dealing well with artifacts.

Correlation between Polar features (already validated in a previous experiment) and Camera HRV features is 0.99 for AVNN, 0.98 for SDNN, 0.78 for rMSSD and 0.87 for pNN50 (outperforming what I could obtain with other optical measures, for example the mio alpha - correlation for rMSSD was only 0.23).

7 - Tips

• use your fingertip
  
• place your fingertip exactly on the camera, you need to cover it all
  
• do touch the camera, but limit pressure. It's important to touch it gently, otherwise too much pressure will impede blood flow (which is what needs to be measured)
  
• try to record using 60 seconds windows. Research on ultra-short term HRV showed that all features can be reliably measured in 50 seconds [3]

Follow @marco_alt

References

[1] Jonathan, E., and Martin Leahy. "Investigating a smartphone imaging unit for photoplethysmography." Physiological Measurement 31.11 (2010): N79.
[2] Voss, A., et al. "Influence of low sampling rate on heart rate variability analysis based on non-linear dynamics." Computers in Cardiology 1995. IEEE, 1995.
[3] Salahuddin, Lizawati, et al. "Ultra short term analysis of heart rate variability for monitoring mental stress in mobile settings." Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE. IEEE, 2007.