# US 11,908,239 B2 · Image recognition network model training method, image recognition method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,239 B2
- Document ID: US11908239B2
- Status: Granted
- Kind: B2
- Application number: 17/319,452
- Filed: 2021-05-13
- Published: 2021-08-26
- Granted: 2024-02-20
- Earliest claimed: 2019-04-02
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Zheng Ge; Ze Qun Jie; Hao Wang; Zhi Feng Li; Di Hong Gong; Wei Liu
- Web page: https://patentgrep.com/patent/US11908239B2

## CPC

- G06V 40/172: {Classification, e.g. identification}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- Y02T 10/40: Engine management systems
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

The disclosure provides an image recognition network model training method, including: acquiring a first image feature corresponding to an image set; acquiring a first identity prediction result by using an identity classifier, and acquiring a first pose prediction result by using a pose classifier; obtaining an identity classifier according to the first identity prediction result and an identity tag, and obtaining a pose classifier according to the first pose prediction result and a pose tag; performing pose transformation on the first image feature by using a generator, to obtain a second image feature corresponding to the image set; acquiring a second identity prediction result by using the identity classifier, and acquiring a second pose prediction result by using the pose classifier; and training the generator.

## Claims

### Claim 1

An image recognition network model training method, performed by a first computer device, the method comprising:
acquiring a first image feature corresponding to an image set, the image set comprising at least one image, an image of the at least one image having an identity tag and a pose tag, the identity tag representing identity information of a target object in the image, and the pose tag representing pose information of the target object in the image;
acquiring, by using an identity classifier, a first identity prediction result corresponding to the first image feature, and acquiring, by using a pose classifier, a first pose prediction result corresponding to the first image feature;
training the identity classifier according to the first identity prediction result and the identity tag, and training the pose classifier according to the first pose prediction result and the pose tag;
performing pose transformation on the first image feature by using a generator, to obtain a second image feature corresponding to the image set, the second image feature being corresponding to a target pose;
acquiring, by using the trained identity classifier, a second identity prediction result corresponding to the second image feature, and acquiring, by using the trained pose classifier, a second pose prediction result corresponding to the second image feature; and
training the generator according to the second identity prediction result, the identity tag, the second pose prediction result, a target pose tag, the second image feature, and a third image feature, the third image feature being an image feature corresponding to an image belonging to the target pose in the image set, the target pose tag representing information of the target pose, and the generator being configured to generate an image recognition network model,
wherein the training the generator comprises:
determining a first minimum loss result according to the second identity prediction result and the identity tag by using a third loss function;
determining a second minimum loss result according to the second pose prediction result and the target pose tag by using a fourth loss function;
determining a third minimum loss result according to the second image feature and the third image feature by using a fifth loss function, the second image feature being a false input of a discriminator, the third image feature being a true input of the discriminator, and the discriminator being configured to discriminate an authenticity of the second image feature and the third image feature;
determining a fourth minimum loss result according to the third image feature by using a sixth loss function; and
training the generator according to the first minimum loss result, the second minimum loss result, the third minimum loss result, and the fourth minimum loss result.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the acquiring the first identity prediction result comprises:
calculating an identity output probability of each identity category according to a network weight value of the identity classifier and the first image feature; and
determining, according to the identity output probability of each identity category, the first identity prediction result corresponding to the first image feature; and
wherein the training the identity classifier comprises:
determining a model parameter of the identity classifier according to the first identity prediction result and the identity tag by using a first loss function.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the calculating the identity output probability comprises:
calculating the identity output probability of each identity category in the following manner:
Y
I
j
=
e
w
I
j
·
E
⁡
(
x
)
∑
i
⁢
e
w
I
i
·
E
⁡
(
x
)
;
where YIj denotes the identity output probability of a jth identity category, i denotes a total quantity of identity categories, E(x) denotes the first image feature, wIj denotes the network weight value of the identity classifier corresponding to the jth identity category, and wIi denotes the network weight value of the identity classifier corresponding to an ith identity category; and
the first loss function is expressed as:
LI=−E[log P(yI|YI)];
where LI denotes the first loss function, E denotes expected value calculation, P(yI|YI) denotes a probability that YI belongs to yI, yI denotes the identity tag, and YI denotes the first identity prediction result.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the acquiring the first pose prediction result comprises:
calculating a pose output probability of each pose category according to a network weight value of the pose classifier and the first image feature; and
determining, according to the pose output probability of each pose category, the first pose prediction result corresponding to the first image feature; and
the training the pose classifier comprises:
determining a model parameter of the pose classifier by using a second loss function according to the first pose prediction result and the pose tag.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the calculating the pose output probability comprises:
calculating the pose output probability of each pose category in the following manner:
Y
P
j
=
e
w
P
j
·
E
⁡
(
x
)
∑
i
⁢
e
w
P
i
·
E
⁡
(
x
)
;
where YPj denotes the pose output probability of a jth pose category, i denotes a total quantity of pose identities, E(x) denotes the first image feature, wPj denotes the network weight value of the pose classifier corresponding to the jth pose category, and wPi denotes the network weight value of the pose classifier corresponding to a ith pose category; and
the second loss function is expressed as:
LP=−E[log P(yP|YP)];
where LP denotes the second loss function, E denotes expected value calculation, P(yP|YP) denotes a probability that YP belongs to yP, yP denotes the pose tag, and YP denotes the second identity prediction result.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the acquiring the second identity prediction result comprises:
acquiring, by using the trained pose classifier and a target pose tag corresponding to the second image feature, the second pose prediction result corresponding to the second image feature; and
acquiring, by using the trained identity classifier and an identity tag corresponding to the first image feature, the second identity prediction result corresponding to the second image feature.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the third loss function is expressed as:
LIG=−E[log P(yI|YIG)];
where LIG denotes the third loss function, E denotes expected value calculation, P(yI|YIG) denotes a probability that YIG belongs to yI, yI denotes the identity tag, and YIG denotes the second identity prediction result; and
wherein the fourth loss function is expressed as:
LPG=−E[log P(yF|YPG)];
where LPG denotes the fourth loss function, (yF|YPG) denotes a probability that YPG belongs to yF, yF denotes the target pose tag, and YPG denotes the second pose prediction result.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the fifth loss function is expressed as:
LadvD=Ef∈Din_fake[D(f)]−Ef∈Din_real[D(f)];
where LadvD denotes the fifth loss function, Din_fake denotes the second image feature, Din_real denotes the third image feature, E denotes expected value calculation, D( ) denotes an input of the discriminator, and f denotes an input feature; and
wherein the sixth loss function is expressed as:
LadvG=−Ef∈Din_fake[D(f)];
where LadvG denotes the sixth loss function.

### Claim 9 (depends on 1)

An image recognition method, performed by a second computer device, the method comprising:
acquiring a first image comprising a first target object, the first target object being corresponding to first pose information;
acquiring, by using the generator in the image recognition network model according to claim 1, a second image corresponding to the first image, the second image comprising a second target object, the second target object being corresponding to second pose information, and the generator being configured to perform pose transformation on an object in an image; and
determining an identity recognition result of the second image by using a recognition module in the image recognition network model.

### Claim 10 (depends on 1)

A face pose transformation method, performed by a third computer device, the method comprising:
acquiring an image corresponding to first pose information, the first pose information representing pose information of the image before frontalization; and
transforming the image by using the generator according to claim 1, to obtain a frontalized face image corresponding to the image, the frontalized face image being corresponding to second pose information, the second pose information representing pose information of the image after frontalization, and the generator being configured to perform pose transformation on an object in an image.

### Claim 11

A model training apparatus, comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
first acquisition code configured to cause at least one of the at least one processor to acquire a first image feature corresponding to an image set, the image set comprising at least one image, an image of the at least one image having an identity tag and a pose tag, the identity tag representing identity information of a target object in the image, and the pose tag representing pose information of the target object in the image;
second acquisition code configured to cause at least one of the at least one processor to acquire, by using an identity classifier, a first identity prediction result corresponding to the first image feature, and acquire, by using a pose classifier, a first pose prediction result corresponding to the first image feature;
first training code configured to cause at least one of the at least one processor to train the identity classifier according to the first identity prediction result and the identity tag, and train the pose classifier according to the first pose prediction result and the pose tag;
transformation code configured to cause at least one of the at least one processor to perform, by using a generator, pose transformation on the first image feature, to obtain a second image feature corresponding to the image set, the second image feature being corresponding to a target pose;
third acquisition code configured to cause at least one of the at least one processor to acquire, by using the trained identity classifier, a second identity prediction result corresponding to the second image feature, and acquire, by using the trained pose classifier, a second pose prediction result corresponding to the second image feature; and
second training code configured to cause at least one of the at least one processor to train the generator according to the second identity prediction result, the identity tag, the second pose prediction result, a target pose tag, the second image feature, and a third image feature, the third image feature being an image feature corresponding to an image belonging to the target pose in the image set, the target pose tag representing information of the target pose, and the generator being configured to generate an image recognition network model,
wherein the second training code comprises:
first minimum loss result determination code configured to cause at least one of the at least one processor to determine a first minimum loss result according to the second identity prediction result and the identity tag by using a third loss function;
second minimum loss result determination code configured to cause at least one of the at least one processor to determine a second minimum loss result according to the second pose prediction result and the target pose tag by using a fourth loss function;
third minimum loss result determination code configured to cause at least one of the at least one processor to determine a third minimum loss result according to the second image feature and the third image feature by using a fifth loss function, the second image feature being a false input of a discriminator, the third image feature being a true input of the discriminator, and the discriminator being configured to discriminate an authenticity of the second image feature and the third image feature;
fourth minimum loss result determination code configured to cause at least one of the at least one processor to determine a fourth minimum loss result according to the third image feature by using a sixth loss function; and
generator training code configured to cause at least one of the at least one processor to train the generator according to the first minimum loss result, the second minimum loss result, the third minimum loss result, and the fourth minimum loss result.

### Claim 12 (depends on 11)

An image recognition apparatus, comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
fourth acquisition code configured to cause at least one of the at least one processor to acquire a first image, the first image comprising a first target object, and the first target object being corresponding to first pose information;
fifth acquisition code configured to cause at least one of the at least one processor to acquire, by using the generator in the image recognition network model according to claim 11, a second image corresponding to the first image, the second image comprising a second target object, the second target object being corresponding to second pose information, and the generator being configured to perform the pose transformation on an object in an image; and
determination code configured to cause at least one of the at least one processor to determine, by using a recognition module in the image recognition network model, an identity recognition result of the second image.

### Claim 13 (depends on 11)

A face pose transformation apparatus, comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
sixth acquisition code configured to cause at least one of the at least one processor to acquire an image, the image being corresponding to first pose information, the first pose information representing pose information of the image before frontalization; and
processing code configured to cause at least one of the at least one processor to transform the image by using the generator according to claim 11, to obtain a frontalized face image corresponding to the image, the frontalized face image being corresponding to second pose information, the second pose information representing pose information of the image after frontalization, and the generator being configured to perform the pose transformation on an object in an image.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the second acquisition code is further configured to cause at least one of the at least one processor to calculate an identity output probability of each identity category according to a network weight value of the identity classifier and the first image feature; and determine, according to the identity output probability of each identity category, the first identity prediction result corresponding to the first image feature; and
wherein the first training code is further configured to cause at least one of the at least one processor to determine a model parameter of the identity classifier according to the first identity prediction result and the identity tag by using a first loss function.

### Claim 15 (depends on 11)

The apparatus according to claim 11, wherein the second acquisition code is further configured to cause at least one of the at least one processor to calculate a pose output probability of each pose category according to a network weight value of the pose classifier and the first image feature; and determine, according to the pose output probability of each pose category, the first pose prediction result corresponding to the first image feature; and
wherein the first training code is further configured to cause at least one of the at least one processor to determine a model parameter of the pose classifier by using a second loss function according to the first pose prediction result and the pose tag.

### Claim 16 (depends on 11)

The apparatus according to claim 11, wherein the third acquisition code is further configured to cause at least one of the at least one processor to acquire, by using the trained pose classifier and a target pose tag corresponding to the second image feature, the second pose prediction result corresponding to the second image feature; and acquire, by using the trained identity classifier and an identity tag corresponding to the first image feature, the second identity prediction result corresponding to the second image feature.

### Claim 17 (depends on 1)

A computer device, comprising: a memory, and a processor,
the memory being configured to store a program; and
the processor being configured to execute the program in the memory, to implement the image recognition network model training method according to claim 1.

### Claim 18 (depends on 1)

A non-transitory computer-readable storage medium, storing a program, the program being executable by at least one processor to perform the method according to claim 1.