# US 2019/0171893 A1 · Face video based heart rate monitoring using pulse signal modelling and tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171893 A1
- Document ID: US20190171893A1
- Status: Published application
- Kind: A1
- Application number: 15/900,788
- Filed: 2018-02-20
- Published: 2019-06-06
- Earliest claimed: 2017-12-05
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Puneet GUPTA; Brojeshwar BHOWMICK; Arpan PAL
- Web page: https://patentgrep.com/patent/US20190171893A1

## CPC

- A61B 5/024: Measuring pulse rate or heart rate
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/02416: {using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence)}
- A61B 5/7221: {Determining signal validity, reliability or quality (preventing, reducing or removing noise induced by motion artefacts A61B5/7207; noise originating from a therapeutic or surgical apparatus A61B5/7217)}
- A61B 5/7257: {using Fourier transforms}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06V 40/168: {Feature extraction; Face representation}
- G06F 2218/08: Feature extraction
- G06F 2218/10: by analysing the shape of a waveform, e.g. extracting parameters relating to peaks
- G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow}
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The present disclosure provides a non-invasive, inexpensive and unobtrusive system that enables heart rate (HR) monitoring by addressing the traditionally known issues with face video based systems due to respiration, facial expressions, out-of-plane movements, camera parameters and environmental factors. These issues are alleviated by filtering, pulse modelling and HR tracking. Quality measures are defined which incorporate out-of-plane movements to define the quality of each video frame unlike existing approaches which provide a single quality for the entire video. To handle out-of-plane movement, Fourier basis function is employed to reconstruct pulse signals at affected locations. Bayesian decision theory based method performs HR tracking using previous HR and quality estimates for improved HR monitoring.

## Claims

### Claim 1

A processor implemented method (200) comprising:
extracting in real time a plurality of non-overlapping windows from an input video with a detected face, wherein the non-overlapping windows correspond to time intervals of the input video, each of the time intervals containing an equal number of frames (202);
analyzing each of the non-overlapping windows (204) by:
extracting a pulse signal from each of the non-overlapping windows (204a);
estimating a first quality measure as a quality of the pulse signal in each of the frames based on out-of-plane movement in a z direction, wherein the first quality measure has (f−1) values for f number of the frames (204b);
estimating a second quality measure based on peak signal to noise ratio (PSNR), wherein the second quality measure is a single value for the pulse signal (204c); and
post processing the pulse signal, to obtain a modelled pulse signal from each of the overlapping windows, by reconstructing the pulse signal at the frames affected by the out-of-plane movements using the remaining frames (204d); and
tracking heart rate corresponding to the modelled pulse signal based on a Bayesian decision theory (206).

### Claim 2 (depends on 1)

The processor implemented method of claim 1, wherein the step of extracting a pulse signal comprises:
identifying a Region of Interest (ROI) corresponding to the detected face in each of the non-overlapping windows, wherein a face area associated with the detected face is divided into non-overlapping square blocks;
performing frame registration between a previous frame and a current frame within each of the non-overlapping windows based on a pre-defined subset of facial landmark points to stabilize location of the ROI in presence of in-plane movement;
extracting temporal signals from each of the non-overlapping square blocks of the ROI using Eulerian method, wherein the temporal signals correspond to motion or color variations in the detected face across time; and
estimating the pulse signal from the extracted temporal signals using a kurtosis based optimization method.

### Claim 3 (depends on 1)

The processor implemented method of claim 1, wherein the step of estimating a first quality measure as a quality of the pulse signal in each of the frames based on out-of-plane movement in a z direction comprises:
calculating a deviation introduced by a change of location of pre-defined facial landmark points in the z direction, wherein the deviation is calculated between each pair of consecutive frames in each of the non-overlapping windows;
estimating the first quality measure by utilizing the pre-defined facial landmark points having a largest deviation in the z direction by:
normalizing the deviations in the pre-defined facial landmark points in the z direction to [0,1]; and
estimating the first quality measure by subtracting the normalized deviations from 1 such that low and high deviations result in high and low values for the first quality measure respectively as
q
^
(
k
-
1
)
=
1
-
(
d
(
k
-
1
)
-
min

(
d
)
max

(
d
)
-
min

(
d
)
)
.

### Claim 4 (depends on 1)

The processor implemented method of claim 1, wherein the post processing of the pulse signal comprises reconstructing the pulse signals at the frames affected by the out-of-plane movements using Fourier basis function based pulse signal modelling, the pulse signal modelling comprising:
decomposing the pulse signal at the frames affected by the out-of-plane movements using the Fourier basis function;
estimating pulse signal modelling parameters by using a weighted least square equation, wherein the pulse signal modelling parameters represent a contribution of the Fourier basis function and weights refer to the first quality measure, wherein solving the weighted least square equation is based on a pseudo inverse equation for minimizing an associated weighted error, and wherein the weighted error is based on the first quality measure and a reconstruction error at each frame; and
obtaining the modelled pulse signal based on the estimated modelling parameters and the Fourier basis function.

### Claim 5 (depends on 1)

The processor implemented method of claim 1, wherein the step of tracking the heart rate comprises:
estimating the heart rate using the Bayesian decision theory by estimating a current heart rate corresponding to a current modelled pulse signal using previous heart rate of a preceding modelled pulse signal and the second quality measure associated with the current and the preceding modelled pulse signals; and
analyzing a spectrum associated with the modelled pulse signal from each of the non-overlapping windows to identify frequency of a maximum amplitude corresponding to the heart rate thereof as an HR frequency.

### Claim 6 (depends on 5)

The processor implemented method of claim 5, wherein using the Bayesian decision theory comprises:
determining a quality of the previous heart rate estimate by utilizing the second quality measure;
calculating a Gaussian distribution of a prior information pertaining to a previously tracked HR and a likelihood function of a current modelled pulse signal by defining a measure of confidence in the previous heart rate estimate along with the determined quality of the previous heart rate estimate;
estimating the prior information required for current heart rate estimation, wherein the prior information is estimated by finding a normalized sum of the Gaussian distribution of the prior information and a parameter θ defined in the HR frequency ranges of 0.7 Hz to 4 Hz;
estimating the likelihood function for determining the current heart rate estimate, wherein the likelihood function is estimated by normalizing the spectrum of the modelled pulse signal corresponding to a current window;
estimating the posterior probability of heart rate frequency by applying Bayes rule, wherein the posterior probability is estimated based on the prior information and the likelihood function; and
estimating a current heart rate frequency, wherein the current heart rate frequency corresponds to the value at which posterior probability is maximum.

### Claim 7

A system (100) comprising:
one or more processors (104); and
one or more internal data storage devices (102) operatively coupled to the one or more processors (104) for storing instructions configured for execution by the one or more processors, the instructions being comprised in:
a window extractor module (102A) configured to:
extract in real time a plurality of non-overlapping windows from an input video with a detected face, wherein the non-overlapping windows correspond to time intervals of the input video, each of the time intervals containing an equal number of frames;
a window analyzer module (102B) configured to:
extract a pulse signal from each of the non-overlapping windows;
estimate a first quality measure as a quality of the pulse signal in each of the frames based on out-of-plane movement in a z direction, wherein the first quality measure has (f−1) values for f number of the frames;
estimate a second quality measure based on peak signal to noise ratio (PSNR), wherein the second quality measure is a single value for the pulse signal; and
post process the pulse signal by reconstructing the pulse signal at the frames affected by the out-of-plane movements using the remaining frames to obtain modelled pulse signals; and
a heart rate (HR) tracker module (102C) configured to:
track heart rate corresponding to the modelled pulse signal from each of the non-overlapping windows based on a Bayesian decision theory.

### Claim 8 (depends on 7)

The system of claim 7, wherein the window analyzer module (102B) is further configured to extract the pulse signal by:
identifying a Region of Interest (ROI) corresponding to the detected face in each of the non-overlapping windows, wherein a face area associated with the detected face is divided into non-overlapping square blocks;
performing frame registration between a previous frame and a current frame within each of the non-overlapping windows based on a pre-defined subset of facial landmark points to stabilize location of the ROI in presence of in-plane movement;
extracting temporal signals from each of the non-overlapping square blocks of the ROI using Eulerian method, wherein the temporal signals correspond to motion or color variations in the detected face across time; and
estimating the pulse signal from the extracted temporal signals using a kurtosis based optimization method.

### Claim 9 (depends on 7)

The system of claim 7, wherein the window analyzer module (102B) is further configured to estimate a first quality measure as a quality of the pulse signal in each of the frames based on out-of-plane movement in a z direction by:
calculating a deviation introduced by a change of location of pre-defined facial landmark points in the z direction, wherein the deviation is calculated between each pair of consecutive frames in each of the non-overlapping windows;
estimating the first quality measure by utilizing the pre-defined facial landmark points having a largest deviation in the z direction by:
normalizing the deviations in the pre-defined facial landmark points in the z direction to [0,1]; and
estimating the first quality measure by subtracting the normalized deviations from 1 such that low and high deviations result in high and low values for the first quality measure respectively as
q
^
(
k
-
1
)
=
1
-
(
d
(
k
-
1
)
-
min

(
d
)
max

(
d
)
-
min

(
d
)
)
.

### Claim 10 (depends on 7)

The system of claim 7, wherein the window analyzer module (102B) is further configured to post process the pulse signal by reconstructing the pulse signals at the frames affected by the out-of-plane movements using Fourier basis function based pulse signal modelling, wherein the pulse signal modelling comprises:
decomposing the pulse signal at the frames affected by the out-of-plane movements using the Fourier basis function;
estimating pulse signal modelling parameters by using a weighted least square equation, wherein the pulse signal modelling parameters represent a contribution of the Fourier basis function and weights refer to the first quality measure, wherein solving the weighted least square equation is based on a pseudo inverse equation for minimizing an associated weighted error, and wherein the weighted error is based on the first quality measure and a reconstruction error at each frame; and
obtaining the modelled pulse signal based on the estimated modelling parameters and the Fourier basis function.

### Claim 11 (depends on 7)

The system of claim 7, wherein the heart rate (HR) tracker module (102C) is further configured to track the heart rate by:
estimating the heart rate using the Bayesian decision theory by estimating a current heart rate corresponding to a current modelled pulse signal using previous heart rate of a preceding modelled pulse signal and the second quality measure associated with the current and the preceding modelled pulse signals; and
analyzing a spectrum associated with the modelled pulse signal from each of the non-overlapping windows to identify frequency of a maximum amplitude corresponding to the heart rate thereof as an HR frequency.

### Claim 12 (depends on 11)

The system of claim 11, wherein the heart rate (HR) tracker module (102C) is further configured to use the Bayesian decision theory by:
determining a quality of the previous heart rate estimate by utilizing the second quality measure;
calculating a Gaussian distribution of a prior information pertaining to a previously tracked HR and a likelihood function of a current modelled pulse signal by defining a measure of confidence in the previous heart rate estimate along with the determined quality of the previous heart rate estimate;
estimating the prior information required for current heart rate estimation, wherein the prior information is estimated by finding a normalized sum of the Gaussian distribution of the prior information and a parameter θ defined in the HR frequency ranges of 0.7 Hz to 4 Hz;
estimating the likelihood function for determining the current heart rate estimate, wherein the likelihood function is estimated by normalizing the spectrum of the modelled pulse signal corresponding to a current window;
estimating the posterior probability of heart rate frequency by applying Bayes rule, wherein the posterior probability is estimated based on the prior information and the likelihood function; and
estimating a current heart rate frequency, wherein the current heart rate frequency corresponds to the value at which posterior probability is maximum.

### Claim 13

A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
extract in real time a plurality of non-overlapping windows from an input video with a detected face, wherein the non-overlapping windows correspond to time intervals of the input video, each of the time intervals containing an equal number of frames;
analyze each of the non-overlapping windows by:
extracting a pulse signal from each of the non-overlapping windows;
estimating a first quality measure as a quality of the pulse signal in each of the frames based on out-of-plane movement in a z direction, wherein the first quality measure has (f−1) values for f number of the frames;
estimating a second quality measure based on peak signal to noise ratio (PSNR), wherein the second quality measure is a single value for the pulse signal; and
post processing the pulse signal, to obtain a modelled pulse signal from each of the overlapping windows, by reconstructing the pulse signal at the frames affected by the out-of-plane movements using the remaining frames; and
track heart rate corresponding to the modelled pulse signal based on a Bayesian decision theory.
