# US 11,308,673 B2 · Using three-dimensional scans of a physical subject to determine positions and/or orientations of skeletal joints in the rigging for a virtual character

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,673 B2
- Document ID: US11308673B2
- Status: Granted
- Kind: B2
- Application number: 16/400,910
- Filed: 2019-05-01
- Published: 2019-11-07
- Granted: 2022-04-19
- Earliest claimed: 2018-05-03
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Sean Michael Comer
- Web page: https://patentgrep.com/patent/US11308673B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 7/11: Region-based segmentation
- G06T 7/73: using feature-based methods
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for using three-dimensional scans of a physical subject to determine positions and/or orientations of skeletal joints in the rigging for a virtual character. At least one articulation segment of a polygon mesh for the virtual character may be determined. The articulation segment may include a subset of vertices in the polygon mesh. An indicator of the position or orientation of the articulation segment of the polygon mesh may be determined. Based on the indicator of the position or orientation of the articulation segment, the position or orientation of at least one joint for deforming the polygon mesh may be determined.

## Claims

### Claim 1

A method comprising:
receiving a plurality of polygon meshes of a digital character, the polygon meshes being representative of a physical subject in a plurality of training poses, each training pose comprising a unique articulation of one or more body parts of the physical subject;
segmenting each of the plurality of polygon meshes corresponding to the plurality of training poses into a plurality of articulation segments;
determining an indicator of position or orientation for each of the articulation segments for each of the polygon meshes corresponding to the plurality of training poses;
determining, based on the indicator of position or orientation of the each of the articulation segments, the position or orientation of one or more joints for the plurality of training poses; and
creating, based on the position or orientation of the one or more joints for the plurality of training poses, a pose space for a pose space deformer to calculate mesh deformations for the digital character by interpolating between the plurality of training poses,
wherein determining an indicator of position or orientation for each of the articulation segments of the polygon mesh comprises determining characteristic axes for each of the articulation segments, a first characteristic axis pointing in the direction of greatest spatial variation between the set of vertices in the articulation segment, a second characteristic axis pointing in the direction of greatest spatial variation between the set of vertices in the articulation segment, subject to the constraint that the second characteristic axis is perpendicular to the first characteristic axis, and a third characteristic axis being perpendicular to both the first characteristic axis and the second characteristic axis.

### Claim 2 (depends on 1)

The method of claim 1, further comprising creating the plurality of polygon meshes by scanning the physical subject in the plurality of training poses.

### Claim 3 (depends on 1)

The method of claim 1, wherein the articulation segments of the polygon mesh consist essentially of vertices which change position or orientation substantially together as a group between different poses amongst the plurality of training poses.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
identifying one of the polygon meshes as a base model; and
performing an alignment operation to align the remaining polygon meshes with the base model.

### Claim 5 (depends on 4)

The method of claim 4, wherein the alignment operation comprises an iterative closest point algorithm.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining an indicator of position or orientation for each of the articulation segments of the polygon mesh comprises determining a local coordinate system for each of the articulation segments.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the local coordinate system for each of the articulation segments comprises determining a center point of the set of vertices in the articulation segment.

### Claim 8 (depends on 7)

The method of claim 7, wherein the center point comprises a centroid point.

### Claim 9 (depends on 6)

The method of claim 6, further comprising comparing the local coordinate system for each of the articulation segments with a reference coordinate system.

### Claim 10 (depends on 1)

The method of claim 1, wherein the characteristic axes comprise orthogonal principal components of the set of vertices in the articulation segment.

### Claim 11 (depends on 10)

The method of claim 10, further comprising determining the orthogonal principal components using eigen-decomposition or Singular Value Decomposition.

### Claim 12 (depends on 1)

The method of claim 1, further comprising determining a rotation matrix for each of the articulation segments which aligns a set of reference axes to the characteristic axes, or vice versa.

### Claim 13 (depends on 1)

The method of claim 1, further comprising loading a file which identifies vertices of each of the articulation segments.

### Claim 14 (depends on 1)

The method of claim 1, further comprising identifying vertices of each of the articulation segments by:
casting a plurality of rays from one of the joints; and
identifying vertices which correspond to polygon faces which are intersected by the rays.

### Claim 15 (depends on 1)

The method of claim 1, wherein the one or more joints are part of a core skeleton for the digital character.

### Claim 16 (depends on 1)

The method of claim 1, further comprising displaying the digital character using a head-mounted, see-through augmented reality display.

### Claim 17 (depends on 16)

The method of claim 16, further comprising displaying the digital character using a plurality of stacked waveguides corresponding to a plurality of depth planes.

### Claim 18 (depends on 1)

The method of claim 1, further comprising using the pose space to train the pose space deformer.

### Claim 19 (depends on 1)

The method of claim 1, further comprising:
inputting one or more joint transformations into the pose space deformer; and
outputting from the pose space deformer a new polygon mesh for the digital character, the new polygon mesh corresponding to a new pose that is not amongst the plurality of poses used to create the pose space.