# US 12,112,418 B2 · Volumetric capture and mesh-tracking based machine learning 4D face/body deformation training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,418 B2
- Document ID: US12112418B2
- Status: Granted
- Kind: B2
- Application number: 18/125,659
- Filed: 2023-03-23
- Published: 2023-07-20
- Granted: 2024-10-08
- Earliest claimed: 2020-03-31
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Kenji Tashiro; Qing Zhang
- Web page: https://patentgrep.com/patent/US12112418B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06N 20/00: Machine learning
- G06T 7/251: {involving models}
- G06T 7/292: Multi-camera tracking
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30201: Face
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Mesh-tracking based dynamic 4D modeling for machine learning deformation training includes: using a volumetric capture system for high-quality 4D scanning, using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence, using mesh registration to establish spatial correspondences between a 4D scanned human face and full-body mesh and a 3D CG physical simulator, and training surface deformation as a delta from the physical simulator using machine learning. The deformation for natural animation is able to be predicted and synthesized using the standard MoCAP animation workflow. Machine learning based deformation synthesis and animation using standard MoCAP animation workflow includes using single-view or multi-view 2D videos of MoCAP actors as input, solving 3D model parameters (3D solving) for animation (deformation not included), and given 3D model parameters solved by 3D solving, predicting 4D surface deformation from ML training.

## Claims

### Claim 1

A method programmed in a non-transitory memory of a device comprising:
using a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously;
using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;
using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator; and
training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.

### Claim 2 (depends on 1)

The method of claim 1 further comprising acquiring multiple, separate 3D scans.

### Claim 3 (depends on 1)

The method of claim 1 further comprising predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
using single-view or multi-view 2D videos of motion capture actors as input;
solving 3D model parameters for animation; and
based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

### Claim 5

An apparatus comprising:
a non-transitory memory for storing an application, the application for:
using a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously;
using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;
using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator; and
training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models; and
a processor coupled to the memory, the processor configured for processing the application.

### Claim 6 (depends on 5)

The apparatus of claim 5 wherein the application is further configured for acquiring multiple, separate 3D scans.

### Claim 7 (depends on 5)

The apparatus of claim 5 wherein the application is further configured for predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

### Claim 8 (depends on 5)

The apparatus of claim 5 wherein the application is further configured for:
using single-view or multi-view 2D videos of motion capture actors as input;
solving 3D model parameters for animation; and
based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

### Claim 9

A system comprising:
a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously; and
a computing device configured for:
using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;
using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator; and
training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.

### Claim 10 (depends on 9)

The system of claim 9 wherein the computing device is further configured for predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

### Claim 11 (depends on 9)

The system of claim 9 wherein the computing device is further configured for:
using single-view or multi-view 2D videos of motion capture actors as input;
solving 3D model parameters for animation; and
based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

### Claim 12

A method programmed in a non-transitory memory of a device comprising:
capturing high quality photos and video simultaneously with a volumetric capture system for high-quality 4D scanning;
performing mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;
performing mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator; and
training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.

### Claim 13 (depends on 12)

The method of claim 12 further comprising acquiring multiple, separate 3D scans.

### Claim 14 (depends on 12)

The method of claim 12 further comprising predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

### Claim 15 (depends on 12)

The method of claim 12 further comprising:
using single-view or multi-view 2D videos of motion capture actors as input;
solving 3D model parameters for animation; and
based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.