# US 10,117,588 B2 · System and method for camera-based heart rate tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,117,588 B2
- Document ID: US10117588B2
- Status: Granted
- Kind: B2
- Application number: 15/923,242
- Filed: 2018-03-16
- Published: 2018-07-19
- Granted: 2018-11-06
- Earliest claimed: 2016-11-14
- Assignees: Nuralogix Corporation
- Applicants: Nuralogix Corporation
- Inventors: Kang Lee; Evgueni Kabakov; Phil Levy
- Web page: https://patentgrep.com/patent/US10117588B2

## CPC

- A61B 5/02427: {Details of sensor}
- A61B 5/024: Measuring pulse rate or heart rate
- A61B 5/02416: {using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence)}
- A61B 5/0255: Recording instruments specially adapted therefor
- A61B 5/026: Measuring blood flow {(A61B3/1233, A61B3/1241 take precedence)}
- A61B 5/0261: {using optical means, e.g. infrared light}
- A61B 5/1455: using optical sensors, e.g. spectral photometrical oximeters
- A61B 5/725: {using specific filters therefor, e.g. Kalman or adaptive filters (specific diagnostics methods using using bioelectric or biomagnetic signals A61B5/316)}
- A61B 5/7253: {characterised by using transforms}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7278: {Artificial waveform generation or derivation, e.g. synthesizing signals from measured signals}
- A61B 5/7485: {Automatic selection of region of interest}
- G06N 20/00: Machine learning
- G06T 7/0016: {involving temporal comparison}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/14546: {for measuring analytes not otherwise provided for, e.g. ions, cytochromes}
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G06T 2207/20081: Training; Learning
- G06T 2207/30104: Vascular flow; Blood flow; Perfusion
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system and method for camera-based heart rate tracking. The method includes: determining bit values from a set of bitplanes in a captured image sequence that represent the HC changes; determining a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes; applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals; applying a Hilbert transform to each of the blood flow data signals; adjusting the blood flow data signals from revolving phase-angles into linear phase segments; determining an instantaneous heart rate for each the blood flow data signals; applying a weighting to each of the instantaneous heart rates; and averaging the weighted instantaneous heart rates.

## Claims

### Claim 1

A method for camera-based heart rate tracking of a human subject, the method comprising:
receiving a captured image sequence of light re-emitted from the skin of the human subject;
determining, using a machine learning model trained with a hemoglobin concentration (HC) changes training set, bit values from a set of bitplanes in the captured image sequence that represent the HC changes of the subject, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known;
determining a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes;
applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals;
applying a Hilbert transform to each of the blood flow data signals;
adjusting the blood flow data signals from revolving phase-angles into linear phase segments;
determining an instantaneous heart rate for each the blood flow data signals;
applying a weighting to each of the instantaneous heart rates;
averaging the weighted instantaneous heart rates; and
outputting the average heart rate.

### Claim 2 (depends on 1)

The method of claim 1, wherein the ROIs are captured from the face of the subject.

### Claim 3 (depends on 1)

The method of claim 1, wherein the ROIs are captured from the wrist or the neck of the subject.

### Claim 4 (depends on 1)

The method of claim 1, wherein the ROIs are non-overlapping.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining a set of bitplanes that maximize SNR comprises:
performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;
identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and
discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

### Claim 6 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a Long Short Term Memory (LSTM) neural network or a non-linear Support Vector Machine.

### Claim 7 (depends on 1)

The method of claim 1, wherein the passband is in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the instantaneous heart rate for each the blood flow data signals comprises applying a differential filter to the linear phase segments to convert the phase-angle data into frequency units representing a count value, the count value for each of the ROIs represents the instantaneous heart rate.

### Claim 9 (depends on 1)

The method of claim 1, further comprising linearizing and differentiating the revolving phase-angles on a phase continuum scale to determine the instantaneous heart rate.

### Claim 10 (depends on 1)

The method of claim 1, wherein the weighting is integrated over an interval in the range of approximately one second to ten seconds.

### Claim 11 (depends on 9)

The method of claim 9, wherein the weighting is integrated over an interval of approximately five seconds.

### Claim 12

A system for camera-based heart rate tracking of a human subject, the system comprising one or more processors and a data storage device, the one or more processors configured to execute:
a TOI module to receive a captured image sequence of light re-emitted from the skin of a human subject, the TOI module determines, using a machine learning model trained with a hemoglobin concentration (HC) changes training set, bit values from a set of bitplanes in the captured image sequence that represent the HC changes of the subject, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known, the TOI module determines a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes;
a filtering module to apply a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals;
a Hilbert transform module to apply a Hilbert transform to each of the blood flow data signals;
an adjustment module to adjust the blood flow data signals from revolving phase-angles into linear phase segments;
a derivative module to determine an instantaneous heart rate for each the blood flow data signals;
a weighting module to apply a weighting to each of the instantaneous heart rates;
a summation module to average the weighted instantaneous heart rates; and
an output module to output the average heart rate.

### Claim 13 (depends on 12)

The system of claim 12, wherein the ROIs are captured from the face of the subject.

### Claim 14 (depends on 12)

The system of claim 12, wherein the ROIs are non-overlapping.

### Claim 15 (depends on 12)

The system of claim 12, wherein the TOI module determines a set of bitplanes that maximize SNR by:
performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;
identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and
discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

### Claim 16 (depends on 12)

The system of claim 12, wherein the passband is in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

### Claim 17 (depends on 12)

The system of claim 12, wherein the derivative module determines the instantaneous heart rate for each the blood flow data signals by applying a differential filter to the linear phase segments to convert the phase-angle data into frequency units representing a count value, the count value for each of the ROIs represents the instantaneous heart rate.

### Claim 18 (depends on 12)

The system of claim 12, wherein the derivative module linearizes and differentiates the revolving phase-angles on a phase continuum scale to determine the instantaneous heart rate.

### Claim 19 (depends on 12)

The system of claim 12, wherein the weighting applied by the weighting module is integrated over an interval in the range of approximately one second to ten seconds.

### Claim 20 (depends on 12)

The system of claim 12, wherein the weighting applied by the weighting module is integrated over an interval of approximately five seconds.