# US 2021/0082097 A1 · Image generation device, image generation method, and storage medium storing program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082097 A1
- Document ID: US20210082097A1
- Status: Published application
- Kind: A1
- Application number: 17/108,284
- Filed: 2020-12-01
- Published: 2021-03-18
- Earliest claimed: 2019-05-10
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Akihiro HAYASAKA
- Web page: https://patentgrep.com/patent/US20210082097A1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06T 1/00: General purpose image data processing
- G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/12: Edge-based segmentation
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04N 1/387: Composing, repositioning or otherwise {geometrically} modifying originals
- G06T 2207/20212: Image combination
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image generation device includes: at least one memory storing a set of instructions; and at least one processor configured to execute the set of instructions to: select a second face image from a plurality of face images stored in advance based on directions of faces included in the plurality of face images and a direction of a face included in an input first face image; deform the second face image based on feature points of the face included in the first face image and feature points of a face included in the second face image such that a face region of the second face image matches a face region of the first face image; and generate a third face image in which the face region of the first face image is synthesized with a region other than the face region of the deformed second face image.

## Claims

### Claim 1

An image processing device comprising:
at least one memory storing a set of instructions; and
at least one processor configured to execute the set of instructions to:
deform a face region of a second face image based on feature points of a face included in an input first face image and feature points of a face included in the second face image such that the face region of the second face image matches a face region of the first face image; and
generate a third face image in which the face region of the first face image is synthesized with a region other than the face region of the deformed second face image.

### Claim 2 (depends on 1)

The image processing device according to claim 1, wherein the at least one processor is configured to select the second face image from a plurality of face images stored in advance based on directions of faces included in the plurality of face images and a direction of the face included in the first face image.

### Claim 3 (depends on 1)

The image processing device according to claim 1, wherein the at least one processor is configured to deform the second face image such that at least the face region of the second face image has a shape to be fitted without nonlinear transformation of the face region of the first face image.

### Claim 4 (depends on 3)

The image processing device according to claim 3, wherein the at least one processor is further configured to:
estimate a geometric transformation parameter for projecting points of the first face image to points of the second face image based on a correspondence relation between the feature points of the first face image and the feature points of the second face image;
project perimeter points on a perimeter of the first face image onto the second face image by using the geometric transformation parameter estimated by the parameter estimation means; and
convert the second face image such that a shape of a region surrounded by a perimeter line of the face region of the second face image and a line, which is formed based on the perimeter points projected onto the second face image, becomes a shape of a region surrounded by a perimeter line of the face region of the first face image and a line formed based on the perimeter points of the first face image.

### Claim 5 (depends on 1)

The image processing device according to claim 1, wherein the at least one processor is configured to select the second face image from face images, in which similarity of values of parameters representing the directions of the faces of the plurality of face images with respect to a value of a parameter representing the direction of the face included in the first face image is within a predetermined reference.

### Claim 6 (depends on 1)

The image generation device according to claim 1, wherein the at least one processor is configured to select the second face image from the plurality of face images based on a distribution of brightness within the face region of the first face image and a distribution of brightness within respective face regions of the plurality of face images.

### Claim 7 (depends on 1)

The image processing device according to claim 1, wherein the at least one processor is configured to select, based on not only the directions of the faces included in the plurality of face images but also a result obtained by comparing directions of faces when the plurality of face images are horizontally reversed with the direction of the face included in the first face image in the selection of the second face image, at least one of the plurality of face images and the plurality of face images horizontally reversed as the second face image.

### Claim 8 (depends on 1)

The image processing device according to claim 1, wherein
the direction of the face is defined by three rotation angles of a rotation angle around right and left axes, a rotation angle around upper and lower axes, and a rotation angle around front and rear axes when a direction of a front-facing face is employed as a reference, and
the at least one processor is configured to select the second face image from the plurality of face images by comparing the rotation angle around right and left axes and the rotation angle around upper and lower axes with each other between the faces included in the plurality of face images and the face included in the first face image.

### Claim 9 (depends on 1)

The image processing device according to claim 1, wherein the face region is a closed region formed by interconnecting feature points positioned on an outside of eyes, a nose, and a mouth and including the eyes, the nose, and the mouth.

### Claim 10

An image processing method comprising:
deforming a face region of a second face image based on feature points of a face included in an input first face image and feature points of a face included in the second face image such that the face region of the second face image matches a face region of the first face image; and
generating a third face image in which the face region of the first face image is synthesized with a region other than the face region of the deformed second face image.

### Claim 11 (depends on 10)

The image processing method according to claim 10, further comprising selecting the second face image from a plurality of face images stored in advance based on directions of faces included in the plurality of face images and a direction of the face included in the first face image.

### Claim 12 (depends on 10)

The image processing method according to claim 10, further comprising deforming the second face image such that at least the face region of the second face image has a shape to be fitted without nonlinear transformation of the face region of the first face image.

### Claim 13 (depends on 12)

The image processing method according to claim 12, further comprising:
estimating a geometric transformation parameter for projecting points of the first face image to points of the second face image based on a correspondence relation between the feature points of the first face image and the feature points of the second face image;
projecting perimeter points on a perimeter of the first face image onto the second face image by using the geometric transformation parameter estimated by the parameter estimation means; and
converting the second face image such that a shape of a region surrounded by a perimeter line of the face region of the second face image and a line, which is formed based on the perimeter points projected onto the second face image, becomes a shape of a region surrounded by a perimeter line of the face region of the first face image and a line formed based on the perimeter points of the first face image.

### Claim 14 (depends on 10)

The image processing method according to claim 10, further comprising selecting the second face image from face images, in which similarity of values of parameters representing the directions of the faces of the plurality of face images with respect to a value of a parameter representing the direction of the face included in the first face image is within a predetermined reference.

### Claim 15 (depends on 10)

The image processing method according to claim 10, further comprising selecting the second face image from the plurality of face images based on a distribution of brightness within the face region of the first face image and a distribution of brightness within respective face regions of the plurality of face images.

### Claim 16 (depends on 10)

The image processing method according to claim 10, further comprising selecting, based on not only the directions of the faces included in the plurality of face images but also a result obtained by comparing directions of faces when the plurality of face images are horizontally reversed with the direction of the face included in the first face image in the selection of the second face image, at least one of the plurality of face images and the plurality of face images horizontally reversed as the second face image.

### Claim 17 (depends on 10)

The image processing method according to claim 10, wherein
the direction of the face is defined by three rotation angles of a rotation angle around right and left axes, a rotation angle around upper and lower axes, and a rotation angle around front and rear axes when a direction of a front-facing face is employed as a reference, and
the method further comprises selecting the second face image from the plurality of face images by comparing the rotation angle around right and left axes and the rotation angle around upper and lower axes with each other between the faces included in the plurality of face images and the face included in the first face image.

### Claim 18 (depends on 10)

The image processing method according to claim 10, wherein the face region is a closed region formed by interconnecting feature points positioned on an outside of eyes, a nose, and a mouth and including the eyes, the nose, and the mouth.

### Claim 19

A non-transitory computer readable storage medium storing a program causing a computer to execute:
image deformation processing of deforming a face region of a second face image based on feature points of a face included in an input first face image and feature points of a face included in the second face image such that the face region of the second face image matches a face region of the first face image; and
image generation processing of generating a third face image in which the face region of the first face image is synthesized with a region other than the face region of the deformed second face image.

### Claim 20 (depends on 19)

The storage medium according to claim 19, the program causing a computer to further execute image selection processing of selecting the second face image from a plurality of face images stored in advance based on directions of faces included in the plurality of face images and a direction of a face included in the first face image.
