# US 2022/0044070 A1 · Methods and systems for automatic generation of massive training data sets from 3d models for training deep learning networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044070 A1
- Document ID: US20220044070A1
- Status: Published application
- Kind: A1
- Application number: 17/309,656
- Filed: 2019-12-17
- Published: 2022-02-10
- Earliest claimed: 2018-11-19
- Assignees: Bodygram, Inc.
- Applicants: Bodygram, Inc.
- Inventors: Chong Jin Koh; Kyohei Kamiyama
- Web page: https://patentgrep.com/patent/US20220044070A1

## CPC

- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06T 7/50: Depth or shape recovery
- G06T 7/60: Analysis of geometric attributes
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 15/20: Perspective computation
- G06T 15/506: {Illumination models}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2210/16: Cloth
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2021: Shape modification
- G06T 2219/2024: Style variation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed are systems and methods for generating large data sets for training deep learning networks (DLNs) for 3D measurements extraction from 2D images taken using a mobile device camera. The method includes the steps of receiving a 3D model of a 3D object; extracting spatial features from the 3D model; generating a first type of augmentation data for the 3D model, such as but not limited to skin color, face contour, hair style, virtual clothing, and/or lighting conditions; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model; generating at least one 2D image from the augmented 3D model by performing a projection of the augmented 3D model onto at least one plane; and generating a training data set to train the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

## Claims

### Claim 1

A computer-implemented method for generating training data sets for training a deep learning network (DLN) for spatial feature extraction from two-dimensional (2D) images of a three-dimensional (3D) object, the computer-implemented method executable by a hardware processor, the method comprising:
receiving a 3D model of the 3D object;
extracting spatial features from the 3D model;
generating a first type of augmentation data for the 3D model;
augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model;
generating at least one 2D image from the augmented 3D model by performing a projection of the augmented 3D model onto at least one plane; and
generating a training data set to train the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D model is a 3D human body model.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the spatial features are one-dimensional (1D) body measurements.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the spatial features are 2D body segments.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the first type of augmentation data is selected from the group consisting of skin color, face contour, hair style, and virtual clothing.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the first type of augmentation data is lighting conditions.

### Claim 8

(canceled)

### Claim 9

(canceled)

### Claim 10

(canceled)

### Claim 11

(canceled)

### Claim 12

(canceled)

### Claim 13

(canceled)

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a second training data set from a second 3D model to train the deep learning network for spatial feature extraction,
wherein the second training data set comprises a second plurality of spatial features, a second plurality of augmented 3D models, and a second plurality of 2D images.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, further comprising:
training the deep learning network for spatial feature extraction using the second training data set.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one 2D image is selected from the group consisting of a front-view image of the 3D model, a side-view image of the 3D model, and an image projected from the 3D model at an angle of 45 degrees with respect to the front-view image.

### Claim 17 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D model is a 3D body model and the method further comprises:
training a sizing machine-learning module, wherein the sizing machine-learning module is trained on ground truth data comprising 1D body measurements extracted from the 3D body model.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein the sizing machine-learning module is trained on ground truth data comprising the 1D body measurements extracted from the 3D body model and one or more user parameters associated with the 3D body model.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the user parameters are selected from the group consisting of a height, a weight, a gender, an age, and a demographic information associated with the user.

### Claim 20 (depends on 17)

The computer-implemented method of claim 17, wherein the sizing machine-learning module comprises a random forest algorithm.

### Claim 21 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D model comprises at least one 3D body model of a fully-nude user or a partially-nude user.

### Claim 22 (depends on 1)

The computer-implemented method of claim 1, wherein the deep learning network (DLN) comprises a convolutional neural network (CNN).

### Claim 23 (depends on 22)

The computer-implemented method of claim 22, wherein the deep learning network (DLN) further comprises a pyramid pooling module.

### Claim 24 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one 3D model is received from a 3D scanner.

### Claim 25

(canceled)

### Claim 26 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training the deep learning network (DLN) for body measurement determination using the training data set.

### Claim 27

(canceled)

### Claim 28

(canceled)

### Claim 29

A computer program product for generating training data sets for training deep learning networks for spatial feature extraction from two-dimensional (2D) images of a three-dimensional (3D) object, comprising a non-transitory computer readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to:
receive a 3D model of the 3D object;
extract spatial features from the 3D model;
generate a first type of augmentation data for the 3D model;
augment the 3D model with the first type of augmentation data to generate an augmented 3D model;
generate at least one 2D image from the augmented 3D model by performing a projection of the augmented 3D model onto at least one plane; and
generate a training data set to train the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

### Claim 30

(canceled)

### Claim 31

(canceled)

### Claim 32

A system for generating training data sets for training deep learning networks for spatial feature extraction from two-dimensional (2D) images of a three-dimensional (3D) object, the system comprising:
at least one processor; and
at least one non-transitory computer readable storage medium having program instructions embodied therein, the program instructions executable by the at least one processor to cause the at least one processor to:
receive a 3D model of the 3D object;
extract spatial features from the 3D model;
generate a first type of augmentation data for the 3D model;
augment the 3D model with the first type of augmentation data to generate an augmented 3D model;
generate at least one 2D image from the augmented 3D model by performing a projection of the augmented 3D model onto at least one plane; and
generate a training data set to train the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.
