# US 12,112,455 B2 · Face-aware offset calculation module and method for facial frame interpolation and enhancement and a face video deblurring system and method using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,455 B2
- Document ID: US12112455B2
- Status: Granted
- Kind: B2
- Application number: 17/202,731
- Filed: 2021-03-16
- Published: 2022-03-03
- Granted: 2024-10-08
- Earliest claimed: 2020-09-01
- Assignees: Vinai Ai Application and Research Joint Stock Co.
- Applicants: Vinai Artificial Intelligence Application and Research Joint Stock Company
- Inventors: Hung Hai Bui; Hoai Minh Nguyen; Phong The Tran; Anh Tuan Tran; Thao Phuong Nguyen Thi
- Web page: https://patentgrep.com/patent/US12112455B2

## CPC

- G06T 5/70: Denoising; Smoothing
- G06F 18/253: {of extracted features}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/73: Deblurring; Sharpening
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided is a Face-aware Offset Calculation (FOC) module and method for facial frame interpolation and enhancement and a face video deblurring system and method using the same. The system comprises: a facial frame enhancement device, including a FOC module, for enhancing a target frame; a facial frame interpolation device, including the FOC module, for interpolating the target frame; and a combination device for combining the enhanced target frame with the interpolated target frame.

## Claims

### Claim 1

A Face-aware Offset Calculation (FOC) method, the method comprising: receiving feature maps and landmark heatmaps of a set of frames; performing a vector concatenation of the feature maps and the landmark heatmaps together to generate inputs, wherein the vector concatenation joins the feature maps and the landmark heatmaps end-to-end by concatenating a vector of the feature maps and a vector of the landmark heatmaps together end-to-end to form a concatenated new vector including elements of the vector of the feature maps and the vector of the landmark heatmaps; and generating an offset map by performing a regression analysis on the inputs, wherein the offset map represents a map of 2D offset vectors that indicate magnitudes by which convolution kernel's sampling locations are to be deformed at each pixel to transform a feature from one frame to another in a deformable convolution network (DCN), and wherein the offset map is configured to be used by the DCN to align and/or interpolate a facial feature map.

### Claim 2 (depends on 1)

A facial frame enhancement method, the method comprising:
extracting feature maps from input frames containing a target frame;
aligning each of the extracted feature maps to the feature map of the target frame based on the offset map generated by using the FOC method according to claim 1;
fusing the aligned feature maps, including the feature map of the target frame, to estimate a feature map of an enhanced target frame; and
reconstructing the enhanced target frame by using the feature map thereof.

### Claim 3 (depends on 2)

The facial frame enhancement method according to claim 2, wherein the input frames include five consecutive frames.

### Claim 4 (depends on 2)

The facial frame enhancement method according to claim 2, wherein the aligning comprising:
obtaining alignment offsets from each of neighboring frames to the target frame with the FOC method; and
aligning, by a deformable convolution layer, each of extracted feature maps of the neighboring frames to the feature map of the target frame using the generated offset map.

### Claim 5 (depends on 2)

The facial frame enhancement method according to claim 2, wherein the aligning is a pyramid, cascading, and deformable (PCD) aligning using the FOC method for offset calculation.

### Claim 6 (depends on 5)

The facial frame enhancement method according to claim 5, wherein a single level aligning in PCD aligning comprising:
using, at each non-bottom level, the feature maps and landmark heatmaps of the downsampled images to calculate the deformable convolution kernels;
refining, at the bottom level, the deformable convolution kernels by fusing them with the upsampled offset estimation from the previous level; and
convolving the deformable kernels with the feature map of the neighboring frame.

### Claim 7 (depends on 5)

The facial frame enhancement method according to claim 5, wherein the number of levels is three.

### Claim 8 (depends on 5)

The facial frame enhancement method according to claim 5, wherein for each non-top level, a simplified FOC method using features of frames as input is used.

### Claim 9 (depends on 1)

A facial frame interpolation method, the method comprising:
extracting feature maps from input frames where a target frame is missing;
interpolating, from each set of extracted feature maps, a feature map of the target frame based on the offset map generated by using the FOC method according to claim 1;
fusing the interpolated feature maps to estimate a combined feature map of the interpolated target frame; and
reconstructing the interpolated target frame by using the combined feature map thereof.

### Claim 10 (depends on 9)

The facial frame interpolation method according to claim 9, wherein the input frames include four frames.

### Claim 11 (depends on 9)

The facial frame interpolation method according to claim 9, wherein the interpolating comprises:
using each set of three neighboring frames from the input frames for the FOC method; and
converting, by a deformable convolution layer, the feature map of center frame in each set to the feature map of target frame using the generated offset map.

### Claim 12 (depends on 9)

The facial frame interpolation method according to claim 9, wherein the interpolating comprises forward interpolating and backward interpolating.

### Claim 13 (depends on 9)

The facial frame interpolation method according to claim 9, wherein the interpolating is a pyramid, cascading, and deformable (PCD) interpolating using the FOC method for offset calculation.

### Claim 14 (depends on 13)

The facial frame interpolation method according to claim 13, wherein a single level interpolating in PCD interpolating comprises:
using, at each non-bottom level, the feature maps and landmark heatmaps of the downsampled images to calculate the deformable convolution kernels;
refining, at the bottom level, the deformable convolution kernels by fusing them with the upsampled offset estimation from the previous level; and
convolving the deformable kernels with the feature map of the neighboring frames.

### Claim 15 (depends on 13)

The facial frame interpolation method according to claim 13, wherein the number of levels is three.

### Claim 16 (depends on 13)

The facial frame interpolation method according to claim 13, wherein for each non-top level, a simplified FOC method using features of frames as input is used.

### Claim 17 (depends on 1)

A face video deblurring method, the method comprising:
extracting feature maps from input frames containing a target frame;
aligning each of the extracted feature maps to the feature map of the target frame based on an offset map generated by using the FOC method according to claim 1;
fusing the aligned feature maps, including the feature map of the target frame, to estimate a feature map of an enhanced target frame;
reconstructing the enhanced target frame by using the feature map thereof;
extracting feature maps from input frames where a target frame is missing;
interpolating, from each set of extracted feature maps, a feature map of the target frame based on an offset map generated by using the FOC method;
fusing the interpolated feature maps to estimate a combined feature map of the interpolated target frame;
reconstructing the interpolated target frame by using the combined feature map thereof; and
combining the enhanced target frame with the interpolated target frame.

### Claim 18 (depends on 17)

The face video deblurring method of claim 17, further comprising pre-processing for removing noise and deblurring some target frames in advance.

### Claim 19

A non-transitory computer-readable medium containing program instructions, which comprises a Face-aware Offset Calculation (FOC) module, executed by a processor or controller, the FOC module of the program instructions when executed by the processor or controller configured to: receive feature maps and landmark heatmaps of a set of frames; perform a vector concatenation of the feature maps and the landmark heatmaps together to generate inputs, wherein the vector concatenation Joins the feature maps and the landmark heatmaps end-to-end by concatenating a vector of the feature maps and a vector of the landmark heatmaps together end-to-end to form a concatenated new vector including elements of the vector of the feature maps and the vector of the landmark heatmaps; and generate an offset map by performing a regression analysis on the inputs, wherein the offset map represents a map of 2D offset vectors that indicate magnitudes by which convolution kernel's sampling locations are to be deformed at each pixel to transform a feature from one frame to another in a deformable convolution network (DCN), and wherein the offset map is configured to be used by the DCN to align and/or interpolate a facial feature map.

### Claim 20 (depends on 19)

A facial frame enhancement device, the device comprising: an extraction module for extracting feature maps from input frames containing a target frame; an alignment module, including the non-transitory computer-readable medium comprised FOC module of claim 19, for aligning each of the extracted feature maps to the feature map of the target frame based on the offset map generated by the non-transitory computer-readable medium comprised FOC module; a fusion module for fusing the aligned feature maps, including the feature map of the target frame, to estimate a feature map of an enhanced target frame; and a reconstruction module for reconstructing the enhanced target frame by using the feature map thereof.

### Claim 21 (depends on 20)

The facial frame enhancement device according to claim 20, wherein the input frames include five consecutive frames.

### Claim 22 (depends on 20)

The facial frame enhancement device according to claim 20, wherein the alignment module is configured to:
obtain alignment offsets from each of neighboring frames to the target frame with the non-transitory computer-readable medium comprised FOC module; and
align, by a deformable convolution layer, each of extracted feature maps of the neighboring frames to the feature map of the target frame using the generated offset map.

### Claim 23 (depends on 20)

The facial frame enhancement device according to claim 20, wherein the alignment module has a pyramid, cascading, and deformable (PCD) architecture using the non-transitory computer-readable medium comprised FOC module for offset calculation.

### Claim 24 (depends on 23)

The facial frame enhancement device according to claim 23, wherein a single level alignment module in PCD architecture is configured to:
use, at each non-bottom level, the feature maps and landmark heatmaps of the downsampled images to calculate the deformable convolution kernels;
refine, at the bottom level, the deformable convolution kernels by fusing them with the upsampled offset estimation from the previous level; and
convolve the deformable kernels with the feature map of the neighboring frame.

### Claim 25 (depends on 23)

The facial frame enhancement device according to claim 23, wherein the number of levels is three.

### Claim 26 (depends on 23)

The facial frame enhancement device according to claim 23, wherein for each non-top level, a simplified FOC module using features of frames as input is used.

### Claim 27 (depends on 19)

A face video deblurring system, the system comprising:
an extraction module for extracting feature maps from input frames containing a target frame;
an alignment module, including the non-transitory computer-readable medium comprised FOC module of claim 19, for aligning each of the extracted feature maps to the feature map of the target frame based on an offset map generated by the non-transitory computer-readable medium comprised FOC module;
a fusion module for fusing the aligned feature maps, including the feature map of the target frame, to estimate a feature map of an enhanced target frame;
a reconstruction module for reconstructing the enhanced target frame by using the feature map thereof an extraction module for extracting feature maps from input frames where a target frame is missing;
an interpolation module, including the non-transitory computer-readable medium comprised FOC module, for interpolating, from each set of extracted feature maps, a feature map of the target frame based on an offset map generated by the non-transitory computer-readable medium comprised FOC module;
a fusion module for fusing the interpolated feature maps to estimate a combined feature map of the interpolated target frame;
a second reconstruction module for reconstructing the interpolated target frame by using the combined feature map thereof;
and a combination device for combining the enhanced target frame with the interpolated target frame.

### Claim 28 (depends on 27)

The face video deblurring system of claim 27, further comprising a pre-processing device for removing noise and deblurring some target frames in advance.

### Claim 29 (depends on 19)

A facial frame interpolation device, the device comprising:
an extraction module for extracting feature maps from input frames where a target frame is missing;
an interpolation module, including the non-transitory computer-readable medium comprised FOC module according to claim 19, for interpolating, from each set of extracted feature maps, a feature map of the target frame based on the offset map generated by the non-transitory computer-readable medium comprised FOC module;
a fusion module for fusing the interpolated feature maps to estimate a combined feature map of the interpolated target frame; and a reconstruction module for reconstructing the interpolated target frame by using the combined feature map thereof.

### Claim 30 (depends on 29)

The facial frame interpolation device according to claim 29, wherein the input frames include four frames.

### Claim 31 (depends on 29)

The facial frame interpolation device according to claim 29, wherein the interpolation module is configured to:
use each setofthreeneghboringframesfromtheinputframesforthenon-transitory computer-readable medium comprised FOC module; and
convert, by a deformable convolution layer, the feature map of center frame in each set to the feature map of target frame using the generated offset map.

### Claim 32 (depends on 29)

The facial frame interpolation device according to claim 29, wherein the interpolation module is configured to perform forward interpolating and backward interpolating.

### Claim 33 (depends on 29)

The facial frame interpolation device according to claim 29, wherein the interpolation module has a pyramid, cascading, and deformable (PCD) architecture using the FOC module for offset calculation.

### Claim 34 (depends on 33)

The facial frame interpolation device according to claim 33, wherein a single level interpolation module in PCD architecture is configured to:
use, at each non-bottom level, the feature maps and landmark heatmaps of the downsampled images to calculate the deformable convolution kernels;
refine, at the bottom level, the deformable convolution kernels by fusing them with the upsampled offset estimation from the previous level; and
convolve the deformable kernels with the feature map of the neighboring frames.

### Claim 35 (depends on 33)

The facial frame interpolation device according to claim 33, wherein the number of levels is three.

### Claim 36 (depends on 33)

The facial frame interpolation device according to claim 33, wherein for each non-top level, a simplified FOC module using features of frames as input is used.