# US 12,213,767 B2 · Video-based method and system for accurately estimating human body heart rate and facial blood volume distribution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,213,767 B2
- Document ID: US12213767B2
- Status: Granted
- Kind: B2
- Application number: 17/696,909
- Filed: 2022-03-17
- Published: 2022-07-14
- Granted: 2025-02-04
- Earliest claimed: 2020-05-25
- Assignees: Zhejiang University
- Applicants: Zhejiang University
- Inventors: Hujun Bao; Xiaogang Xu; Xiaolong Wang
- Web page: https://patentgrep.com/patent/US12213767B2

## CPC

- A61B 5/02416: {using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence)}
- A61B 5/0295: using plethysmography, i.e. measuring the variations in the volume of a body part as modified by the circulation of blood therethrough, e.g. impedance plethysmography
- A61B 5/7232: {involving compression of the physiological signal, e.g. to extend the signal recording period}
- A61B 5/725: {using specific filters therefor, e.g. Kalman or adaptive filters (specific diagnostics methods using using bioelectric or biomagnetic signals A61B5/316)}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0016: {involving temporal comparison}
- G06T 7/11: Region-based segmentation
- G06T 7/73: using feature-based methods
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30076: Plethysmography
- G06T 2207/30088: Skin; Dermal
- G06T 2207/30201: Face
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Provided is a video-based method and system for accurately estimating heart rate and facial blood volume distribution, and the method mainly comprises the following steps: firstly, carrying out face detection of video frame containing human face, and extracting face image sequence and face key position points sequence in time dimension; secondly, compressing these sequence of face image and face key position points to obtain the facial signals in time dimension; thirdly, estimating facial blood volume distribution by facial signals mentioned in third step; finally, estimating heart rate values by using model based on deep learning technology and the spectrum analysis method respectively, then fusing the estimation results by Kalman filter to promote the accuracy of heart rate estimation.

## Claims

### Claim 1

A video-based method for accurately estimating a human heart rate and facial blood volume distribution, comprising the following steps:
(1) detecting a human face region in video frame, and extracting a human face image sequence and face key position points in time dimension, extracting a global face signal and a set of face roi signals based on the face image sequence, preprocessing the signals;
wherein the step (1) specifically comprises:
(1.1) using a convolution neural network model to detect the human face region and the face position key points in the video frame, and respectively generating a human face image sequence and a face key position point sequence in time dimension;
(1.2) extracting the global face signal and the set of the face roi signals, respectively, based on the face image sequence, the global face signal can be extracted as shown by Formula 3, where: face_sig is a compressed signal, PCompress ( ) is a compression function which is used to calculate an average pixel intensity of a face image of the face image sequence, and face_seq is the face image sequence;
face_sig=PCompress(face_seq) (3)
segmenting the face image by roi blocks with R×R size to obtain roi block image sequences in time dimension, as shown in Formula 4, where: face_roii represents an ith roi block image sequence, face_roi_seq is a set of roi block image sequences, and mxn is a sum of the roi blocks;
face_roi_seq={face_roi1,face_roi2, . . . ,face_roii, . . . ,face_roim×n} (4)
compressing each roi block image sequence, as shown in Formula5, where: face_roi_seq is the set of roi block image sequences, PCompress ( ) is the compression function for calculating mean of pixel intensity of the image of the sequence, and face_roi_sig is the result of PCompress ( );
face_roi_sig=PCompress(face_roi_seq) (5)
where:
face_roi_sig={face_roi_sig1, . . . ,face_roi_sigi, . . . ,face_roi_sigm×n} (6)
in Formula 6, face_roi_sigi is a signal compressed by the ith roi block image sequence, and m×n is the sum of the roi blocks;
(1.3) preprocessing the global face signal and the set of the face roi signals to eliminate components outside a specified frequency range;
(2) estimate heart rate value and facial blood volume distribution based on a reference signal and the set of roi signals;
(3) estimate heart rate value based on heart rate distribution probability by using a heart rate estimation model based on Long and Short Time Memory Network (LSTM) and a residual convolution neural network model;
(4) fusing results of the heart rate value of the step (2) and the step (3) based on Kalman filtering.

### Claim 2 (depends on 1)

The video-based method for accurately estimating a human heart rate and a facial blood volume distribution according to claim 1, wherein the step (2) specifically comprises:
(2.1) calculating the reference signal by linear weighting, as shown in Formula 9, where sig_ref is the reference signal, roi_sig_r is the preprocessed set of the face roi signals, and m×n is the sum of the roi blocks;
sig_ref
=
weight_set
×
roi_sig
⁢
_r
=
∑
i
=
1
m
×
n
⁢
w
i
×
roi_sig
⁢
_r
i
⁢
⁢
weight_set
=
{
w
1
,
w
2
,
…
⁢
,
w
i
,
…
⁢
,
w
m
×
n
}
(
9
)
roi_sig
⁢
_r
=
sig
⁢
⁢
process
⁡
(
face_roi
⁢
_sig
)
(
8
)
where: weight_set is a calculated weight set; sigprocess ( ) is a signal preprocessing function;
(2.2) calculating a frequency spectrum of the reference signal by using a lomb-scargle spectrum analysis method, the heart rate value corresponds to a extremum value of the frequency spectrum;
(2.3) estimating the facial blood volume distribution.

### Claim 3 (depends on 2)

The video-based method for accurately estimating human heart rate and a facial blood volume distribution according to claim 2, wherein the step (2.3) is specifically as below:
as shown in Formula 13, sig_ref_sd is the spectrum of the reference signal, and v is the blood volume distribution;
v=Volume(sig_ref_sd) (13)
where, Volume ( ) is a function for calculating the blood volume distribution, a specific form of which is shown in Formula 14;
Volume
(
)
=
[
fs
ref
⊗
fs
roi
(
1
,
1
)
fs
ref
⊗
fs
roi
(
1
,
2
)
…
fs
ref
⊗
fs
roi
(
1
,
n
)
fs
ref
⊗
fs
roi
(
2
,
1
)
fs
ref
⊗
fs
roi
(
2
,
2
)
…
fs
ref
⊗
fs
roi
(
2
,
n
)
⋮
⋮
⋱
⋮
fs
ref
⊗
fs
roi
(
m
,
1
)
fs
ref
⊗
fs
roi
(
m
,
2
)
…
fs
ref
⊗
fs
roi
(
m
,
n
)
]
(
14
)
in Formula 14, fsref is a frequency spectrum of the reference signal, fsroi is a frequency spectrum of the roi signal, ⊗ is a convolution operator, and m and n are sum of the roi blocks in horizontal and vertical directions, respectively.

### Claim 4 (depends on 1)

The video-based method for accurately estimating human heart rate and facial blood volume distribution according to claim 1, wherein in step (3), a training method of the heart rate estimation model constructed based on the LSTM and the residual convolution neural network model is as follows:
(3.1) making training samples
forming a key point sequence in time dimension based on the face key position points extracted in step (1), selecting an image sequence formed by forehead, left and right cheek regions in time dimension based on the face key position points, and compressing the selected critical images to construct training samples;
(3.2) normalizing train samples to obtain the normalized train samples sig_nor;
(3.3) constructing a heart rate estimation module based on the LSTM;
wherein the module includes two network structures of One-dimension Convolutional Neural Network (1D-CNN) and LSTM; firstly, the sig_nor signal obtained in step (3.2) is used as the input data of this module, and preliminary features corresponding to the sig_nor signal are extracted based on the 1D-CNN sub-module, on this basis, LSTM sub-module is used to extract features; finally, fusing feature vectors of various stage output of the LSTM by using an attention mechanism, then obtain the highest level feature vector expressed as featurelstm,
(3.4) constructing the heart rate estimation module based on Resnet;
wherein the module extracts features of waveform distribution of the signal based on Resnet, and takes the sig_nor as an input sample of the module, then obtain the highest level feature vector expressed as featureresnet;
(3.5) fusing the modules in steps (3.3) and (3.4) to construct the heart rate estimation multi-model;
combine output features of the modules in step (3.3) and step (3.4) into feature vector, then estimate heart rate value by the fully connected network (FCN);
wherein a basic estimation process is shown in Formula 21, where: res_pro is a model estimation result vector, FCN ( ) is a fully connected layer, and Concat ( ) is a vector combining function;
res_pro=FCN(Concat(featurelstm,featureresnet)) (21)
on this basis, the heart rate value is estimated, and a basic process of extracting the heart rate value is shown in Formula 22, where: heart_rate_pre is a heart rate estimation value, mean ( ) is a mean function, and max_reg ( ) is a function for searching for a heart rate range corresponding to a maximum probability value;
heart_rate_pre=mean(max_reg(res_pro)) (22)

### Claim 5 (depends on 1)

The video-based method for accurately estimating human heart rate and facial blood volume distribution according to claim 1, wherein the step (4) specifically comprises:
fusing the heart rate values obtained by two estimating methods shown in Formulas 25 and 26,
x
k
′
=
x
k
+
K
⁡
(
z
k
-
Hx
k
)
(
25
)
K
=
P
k
⁢
H
T
HP
k
⁢
H
T
+
R
k
(
26
)
where xk′ is a fused heart rate value, xk is the heart rate estimation value obtained in step (3), zk is the heart rate value obtained in step (2), K is a fusion coefficient, Pk is a predicted variance, Rk is a measured variance, and H and HT represents a relation matrix and a transposition form thereof, respectively.