# US 11,908,083 B2 · Deforming custom mesh based on body mesh

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,083 B2
- Document ID: US11908083B2
- Status: Granted
- Kind: B2
- Application number: 17/446,533
- Filed: 2021-08-31
- Published: 2023-03-02
- Granted: 2024-02-20
- Earliest claimed: 2021-08-31
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Yanli Zhao; Matan Zohar; Brian Fulkerson; Georgios Papandreou; Haoyang Wang
- Web page: https://patentgrep.com/patent/US11908083B2

## CPC

- G06T 17/205: {Re-meshing}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04L 51/046: Interoperability with other network applications or services
- H04L 51/10: Multimedia information
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2210/16: Cloth
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

## Claims

### Claim 1

A method comprising:
receiving, by one or more processors of a client device, a video that includes a depiction of a real-world object;
generating, by the one or more processors, a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video;
obtaining an external mesh associated with an augmented reality (AR) element;
automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh;
deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and
modifying the video to include a display of the AR element based on the deformed external mesh.

### Claim 2 (depends on 1)

The method of claim 1, wherein the real-world object comprises a person, and wherein the 3D body mesh represents a whole-body of the person depicted in the video.

### Claim 3 (depends on 2)

The method of claim 2, wherein the augmented reality element comprises a fashion item.

### Claim 4 (depends on 1)

The method of claim 1, further comprising adding the augmented reality element to the video at a position determined based on the established correspondence between the 3D body mesh and the external mesh.

### Claim 5 (depends on 1)

The method of claim 1, wherein deforming the external mesh comprises:
computing a change in body shape, body state or body properties of the real-world object across the frames of the video; and
adjusting a shape of the external mesh based on the change in the shape, body state or body properties.

### Claim 6 (depends on 1)

The method of claim 1, wherein deforming the external mesh comprises:
computing a change in position or scale of the real-world object across the frames of the video; and
adjusting a position or scale of the external mesh based on the change in the position or scale.

### Claim 7 (depends on 1)

The method of claim 1, wherein deforming the external mesh comprises:
computing a change in rotation of the real-world object across the frames of the video; and
rotating the external mesh in 3D based on the change in the rotation.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining a first set of coordinates of the 3D body mesh in a normal-tangent space for a first frame of the frames of the video;
determining a second set of coordinates of the 3D body mesh in the normal-tangent space for a second frame of the frames of the video;
computing, in real time, a change between the first and second sets of coordinates in the normal-tangent space, and
transferring the change between the first and second sets of coordinates in the normal-tangent space to the external mesh.

### Claim 9 (depends on 8)

The method of claim 8, wherein a 3D position and a 3D orientation of the external mesh are adjusted based on the change between the first and second sets of coordinates in the normal-tangent space.

### Claim 10 (depends on 1)

The method of claim 1, wherein the external mesh is deformed without using a rig of the real-world object.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
receiving, by a messaging application, placement information for the external mesh from a third-party designer of the external mesh; and
adding the external mesh to the 3D body mesh based on the placement information.

### Claim 12 (depends on 11)

The method of claim 11, wherein the placement information comprises one or more proximity parameters relative to the 3D body mesh.

### Claim 13 (depends on 12)

The method of claim 12, wherein the 3D body mesh comprises a plurality of portions, and wherein the one or more proximity parameters indicate a distance between a given edge of the external mesh to a given set of vertices of one of the plurality of portions, further comprising:
adding the augmented reality element to the video at a position that corresponds to the distance between the given edge of the external mesh and the given set of vertices; and
maintaining the distance between the given edge of the external mesh and the given set of vertices throughout the frames in response to deforming the external mesh.

### Claim 14 (depends on 13)

The method of claim 13, wherein the distance comprises a minimum distance that is maintained between the given edge of the external mesh and the given set of vertices.

### Claim 15 (depends on 13)

The method of claim 13, wherein the distance comprises a maximum distance that is maintained between the given edge of the external mesh and the given set of vertices.

### Claim 16 (depends on 11)

The method of claim 11, wherein the placement information comprises relative UV channel coordinates of the body mesh, further comprising:
computing UV coordinates in a UV channel associated with the body mesh;
computing a set of UV coordinates for the external mesh based on the UV coordinates associated with the body mesh and the relative UV channel coordinates in the placement information; and
adding the AR element to the video at a position that corresponds the set of UV coordinates.

### Claim 17 (depends on 1)

The method of claim 1, further comprising generating the external mesh by:
causing a reference body model to be presented on a given client device associated with an augmented reality element designer, the reference body model being different from the real-world object and the 3D body mesh;
receiving input from the AR element designer that places the AR element in relation to the reference body model; and
computing placement correspondence information for the AR element based on the placement of the AR element in relation to the reference body model; and
generating the external mesh based on the computed placement correspondence information.

### Claim 18 (depends on 1)

The method of claim 1, wherein the AR element comprises an AR fashion item, further comprising:
fitting or placing the AR fashion item over the real-world object based on the external mesh; and
adjusting a pose, shape, and scale of the AR fashion item, based on the deformed external mesh associated with the AR fashion item, in real time based on movement of the real-world object across the frames of the video.

### Claim 19

A system comprising:
a processor of a client device; and
a memory component having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
receiving a video that includes a depiction of a real-world object;
generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video;
obtaining an external mesh associated with an augmented reality (AR) element;
automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh;
deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and
modifying the video to include a display of the AR element based on the deformed external mesh.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by a processor of a client device, cause the processor to perform operations comprising:
receiving a video that includes a depiction of a real-world object;
generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video;
obtaining an external mesh associated with an augmented reality (AR) element;
automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh;
deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and
modifying the video to include a display of the AR element based on the deformed external mesh.