# US 12,112,571 B2 · Neural network-based key point training apparatus and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,571 B2
- Document ID: US12112571B2
- Status: Granted
- Kind: B2
- Application number: 17/762,819
- Filed: 2020-12-01
- Published: 2022-10-27
- Granted: 2024-10-08
- Earliest claimed: 2020-07-01
- Assignees: Deepbrain Ai Inc.
- Applicants: Deepbrain Ai Inc.
- Inventors: Gyeongsu Chae; Guembuel Hwang
- Web page: https://patentgrep.com/patent/US12112571B2

## CPC

- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A neural network-based key point training apparatus according to an embodiment disclosed includes a key point model trained to extract key points from an input image and an image reconstruction model trained to reconstruct the input image with the key points output by the key point model as the input.

## Claims

### Claim 1

A neural network-based key point training apparatus comprising:
a key point model trained to extract key points from an input image; and
an image reconstruction model trained to reconstruct the input image with the key points output by the key point model as the input,
wherein the key point model comprises:
an extraction module trained to extract a feature vector from the input image; and
a prediction module trained to predict key point coordinates of the input image based on the feature vector extracted by the extraction module; and
the image reconstruction model is trained to reconstruct the input image based on the key point coordinates output by the key point model,
wherein optimized parameters of the key point model and the image reconstruction model are calculated through the following equation:
θ*,φ*=argminθ,φ(αLprediction+βLreconstruction) (Equation)
where, θ*: optimized parameter of key point model;
φ*: optimized parameter of image reconstruction model;
Lprediction: objective function of key point model,
Lprediction=∥K−G(I;θ)∥;
K: labelled key point coordinates of input image;
G: neural network constituting key point model;
θ: parameter of the key point model;
I: input image;
G(I;θ): key point coordinates predicted from key point model;
∥K−G(I;θ)∥: function to find difference between key point coordinates predicted from key point model and labelled key point coordinates of input image;
Lreconstruction: objective function of image reconstruction model, Lreconstruction=∥I−D(G(I;θ);φ)∥;
D: neural network constituting image reconstruction model;
φ: parameter of image reconstruction model;
∥I−D(G(I;θ);φ)∥: function to find difference between input image and image reconstructed by image reconstruction model;
α: weight of objective function of key point model; and
β: weight of objective function of image reconstruction model.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the key point model is trained to minimize differences between the extracted key points and key points labelled with respect to the input image; and
the image reconstruction model is trained to minimize a difference between the reconstructed image and the input image.

### Claim 3 (depends on 2)

The apparatus according to claim 2, wherein the key point model is primarily trained to minimize the differences between the extracted key points and the key points labelled with respect to the input image and is secondarily trained to extract the key point so that a difference between the reconstructed image and the input image is minimized.

### Claim 4

A neural network-based key point training apparatus comprising:
a key point model trained to extract key points from an input image; and
an image reconstruction model trained to reconstruct the input image with the key points output by the key point model as the input,
wherein the key point model comprises:
an extraction module trained to extract a feature tensor from the input image; and
a prediction module trained to predict a key point image based on the feature tensor extracted by the extraction module; and
the image reconstruction model is trained to reconstruct the input image based on the key point image output by the key point model,
wherein the key point image is an image indicating whether or not each pixel corresponds to the key point in an image space corresponding to the input image as a probability value,
wherein an objective function Lprediction of the key point model is expressed through the following equation:
Lprediction=−Σ{ptarget*xi,yi)log(p(xi,yi))+(1−ptarget(xi,yi))log(1−p(xi,yi))} (Equation)
where, p(xi,yi): probability value of whether or not pixel p(xi, yi) is key point;
p(xi,yi)=probability distribution (P(F(xi,yi); δ));
P: neural network constituting key point model;
F(xi,yi): feature tensor of pixel (xi, yi);
δ: parameter of key point model; and
ptarget(xi,yi): value of whether or not pixel (xi,yi) of the input image is labelled key point,
wherein optimized parameters of the key point model and the image reconstruction model are calculated through the following equation:
δ*,η*=argminδ,η(αLprediction+βLreconstruction) (Equation)
where, δ*: optimized parameter of key point model;
η*: optimized parameter of image reconstruction model;
α: weight of objective function of key point model;
β: weight of objective function of image reconstruction model;
Lreconstruction: objective function of the image reconstruction model, Lreconstruction=∥I−H(P(I;δ);η)∥;
I: input image;
H: neural network constituting image reconstruction model; and
η: parameters of image reconstruction model.