# US 2023/0054515 A1 · Image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0054515 A1
- Document ID: US20230054515A1
- Status: Published application
- Kind: A1
- Application number: 17/977,682
- Filed: 2022-10-31
- Published: 2023-02-23
- Earliest claimed: 2021-01-22
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Jinlong PENG; Keke HE; Zhengkai JIANG; Yang YI; Xiaoming YU; Juanhui TU; Yi ZHOU; Chenghao LIU; Yabiao WANG; Ying TAI; Chengjie WANG; Jilin LI; Feiyue HUANG
- Web page: https://patentgrep.com/patent/US20230054515A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/803: {of input or preprocessed data}
- G06V 10/82: using neural networks
- G06V 40/193: {Preprocessing; Feature extraction}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product. The image gaze correction method includes: acquiring a to-be-corrected eye image from a to-be-corrected image, generating, based on the to-be-corrected eye image, an eye motion flow field and an eye contour mask, the eye motion flow field being used for adjusting a pixel position in the to-be-corrected eye image, and the eye contour mask being used for indicating a probability that the pixel position in the to-be-corrected eye image belongs to an eye region, performing, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image, and generating a gaze corrected image based on the corrected eye image.

## Claims

### Claim 1

An image gaze correction method, performed by an electronic device, and the image gaze correction method comprising:
acquiring a to-be-corrected eye image from a to-be-corrected image;
generating, based on the to-be-corrected eye image, an eye motion flow field and an eye contour mask, the eye motion flow field being used for adjusting a pixel position in the to-be-corrected eye image, and the eye contour mask being used for indicating a probability that the pixel position in the to-be-corrected eye image belongs to an eye region;
performing, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image; and
generating a gaze corrected image based on the corrected eye image.

### Claim 2 (depends on 1)

The image gaze correction method according to claim 1, wherein the performing comprises:
performing, by using the eye motion flow field, transformation processing on the to-be-corrected eye image to obtain an initially corrected eye image; and
performing, by using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a further corrected eye image, and determining the further corrected eye image as the corrected eye image, wherein the gaze correction processing comprises the transformation processing and the adjustment processing.

### Claim 3 (depends on 2)

The image gaze correction method according to claim 2, wherein the performing, by using the eye contour mask, adjustment processing on the initially corrected eye image comprises:
fusing pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image;
fusing pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the to-be-corrected eye image to obtain a second intermediate image; and
integrating pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the further corrected eye image.

### Claim 4 (depends on 1)

The image gaze correction method according to claim 1, wherein the generating comprises:
performing down-sampling processing on the to-be-corrected eye image to generate a down-sampled image; and
performing up-sampling processing on the down-sampled image to obtain the eye motion flow field and the eye contour mask, wherein during the up-sampling processing, a feature map generated by the current up-sampling operation and a feature map of a same size generated during the down-sampling processing are cascaded and then used as an input image of a next up-sampling operation.

### Claim 5 (depends on 4)

The image gaze correction method according to claim 4, wherein the down-sampling processing and the up-sampling processing are implemented by a gaze correction model, the eye motion flow field is data of a first channel and a second channel in output data of the gaze correction model, and the eye contour mask is data of a third channel in the output data.

### Claim 6 (depends on 1)

The image gaze correction method according to claim 1, wherein the acquiring comprises:
recognizing contour key points of eyes from the to-be-corrected image;
determining a minimum circumscribed rectangle based on the contour key points;
specifying a multiple for extension of the minimum circumscribed rectangle to obtain an image capture frame; and
capturing, based on the image capture frame, the to-be-corrected eye image from the to-be-corrected image.

### Claim 7 (depends on 1)

The image gaze correction method according to claim 1, wherein the generating comprises:
integrating the corrected eye image into an image capture frame position of the to-be-corrected image to obtain an integrated image, wherein the image capture frame position is a position of the to-be-corrected eye image in the to-be-corrected image; and
performing image harmonization processing at the image capture frame position in the integrated image to obtain the gaze corrected image, wherein the image harmonization process is used for eliminating boundary traces at the image capture frame position.

### Claim 8 (depends on 7)

The image gaze correction method according to claim 7, wherein the performing image harmonization processing at the image capture frame position in the integrated image comprises:
generating an initialized mask image of the same size as the to-be-corrected image, wherein a pixel value of the initialized mask image at the image capture frame position is a first specified value, pixel values of remaining positions are a second specified value different from the first specified value, and the remaining positions are positions in the initialized mask image except the image capture frame position;
performing noise processing on the initialized mask image to obtain a processed mask image;
fusing pixel values of corresponding positions in the processed mask image and the integrated image to obtain a first generated image;
fusing pixel values of corresponding positions in a mapped image corresponding to the processed mask image and the to-be-corrected image to obtain a second generated image; and
integrating pixel values of corresponding positions in the first generated image and the second generated image to obtain the gaze corrected image.

### Claim 9

An image gaze correction apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
eye image acquisition code configured to cause the at least one processor to acquire a to-be-corrected eye image from a to-be-corrected image;
flow field mask generation code configured to cause the at least one processor to generate, based on the to-be-corrected eye image, an eye motion flow field and an eye contour mask, the eye motion flow field being used for adjusting a pixel position in the to-be-corrected eye image, and the eye contour mask being used for indicating a probability that the pixel position in the to-be-corrected eye image belongs to an eye region;
gaze correction processing code configured to cause the at least one processor to perform, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image; and
eye image integration code configured to cause the at least one processor to generate a gaze corrected image based on the corrected eye image.

### Claim 10 (depends on 9)

The image gaze correction apparatus according to claim 9, wherein the gaze correction processing code is further configured to cause the at least one processor to:
perform, by using the eye motion flow field, transformation processing on the to-be-corrected eye image to obtain an initially corrected eye image; and
perform, by using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a further corrected eye image, and determining the further corrected eye image as the corrected eye image, wherein the gaze correction processing comprises the transformation processing and the adjustment processing.

### Claim 11 (depends on 10)

The image gaze correction apparatus according to claim 10, wherein the gaze correction processing code is further configured to cause the at least one processor to:
fuse pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image;
fuse pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the to-be-corrected eye image to obtain a second intermediate image; and
integrate pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the further corrected eye image.

### Claim 12 (depends on 9)

The image gaze correction apparatus according to claim 9, wherein the flow field mask generation code is further configured to cause the at least one processor to:
perform down-sampling processing on the to-be-corrected eye image to generate a down-sampled image; and
perform up-sampling processing on the down-sampled image to obtain the eye motion flow field and the eye contour mask, wherein during the up-sampling processing, a feature map generated by the current up-sampling operation and a feature map of a same size generated during the down-sampling processing are cascaded and then used as an input image of a next up-sampling operation.

### Claim 13 (depends on 12)

The image gaze correction apparatus according to claim 12, wherein the down-sampling processing and the up-sampling processing are implemented by a gaze correction model, the eye motion flow field is data of a first channel and a second channel in output data of the gaze correction model, and the eye contour mask is data of a third channel in the output data.

### Claim 14 (depends on 9)

The image gaze correction apparatus according to claim 9, wherein the eye image acquisition code is further configured to cause the at least one processor to:
recognize contour key points of eyes from the to-be-corrected image;
determine a minimum circumscribed rectangle based on the contour key points;
specify a multiple for extension of the minimum circumscribed rectangle to obtain an image capture frame; and
capture, based on the image capture frame, the to-be-corrected eye image from the to-be-corrected image.

### Claim 15 (depends on 9)

The image gaze correction apparatus according to claim 9, wherein the eye image integration code is further configured to cause the at least one processor to:
integrate the corrected eye image into an image capture frame position of the to-be-corrected image to obtain an integrated image, wherein the image capture frame position is a position of the to-be-corrected eye image in the to-be-corrected image; and
perform image harmonization processing at the image capture frame position in the integrated image to obtain the gaze corrected image, wherein the image harmonization process is used for eliminating boundary traces at the image capture frame position.

### Claim 16 (depends on 15)

The image gaze correction apparatus according to claim 15, wherein the perform image harmonization processing at the image capture frame position in the integrated image comprises:
generating an initialized mask image of the same size as the to-be-corrected image, wherein a pixel value of the initialized mask image at the image capture frame position is a first specified value, pixel values of remaining positions are a second specified value different from the first specified value, and the remaining positions are positions in the initialized mask image except the image capture frame position;
performing noise processing on the initialized mask image to obtain a processed mask image;
fusing pixel values of corresponding positions in the processed mask image and the integrated image to obtain a first generated image;
fusing pixel values of corresponding positions in a mapped image corresponding to the processed mask image and the to-be-corrected image to obtain a second generated image; and
integrating pixel values of corresponding positions in the first generated image and the second generated image to obtain the gaze corrected image.

### Claim 17

A non-transitory computer-readable storage medium, storing computer code that when executed by at least one processor causes the at least one processor to:
acquire a to-be-corrected eye image from a to-be-corrected image;
generate, based on the to-be-corrected eye image, an eye motion flow field and an eye contour mask, the eye motion flow field being used for adjusting a pixel position in the to-be-corrected eye image, and the eye contour mask being used for indicating a probability that the pixel position in the to-be-corrected eye image belongs to an eye region;
perform, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image; and
generate a gaze corrected image based on the corrected eye image.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the perform comprises:
performing, by using the eye motion flow field, transformation processing on the to-be-corrected eye image to obtain an initially corrected eye image; and
performing, by using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a further corrected eye image, and determining the further corrected eye image as the corrected eye image, wherein the gaze correction processing comprises the transformation processing and the adjustment processing.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the performing, by using the eye contour mask, adjustment processing on the initially corrected eye image comprises:
fusing pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image;
fusing pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the to-be-corrected eye image to obtain a second intermediate image; and
integrating pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the further corrected eye image.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the generate comprises:
performing down-sampling processing on the to-be-corrected eye image to generate a down-sampled image; and
performing up-sampling processing on the down-sampled image to obtain the eye motion flow field and the eye contour mask, wherein during the up-sampling processing, a feature map generated by the current up-sampling operation and a feature map of a same size generated during the down-sampling processing are cascaded and then used as an input image of a next up-sampling operation.
