# US 2025/0095308 A1 · Deforming real-world object using an external mesh

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095308 A1
- Document ID: US20250095308A1
- Status: Published application
- Kind: A1
- Application number: 18/969,689
- Filed: 2024-12-05
- Published: 2025-03-20
- Earliest claimed: 2021-09-20
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Matan Zohar; Yanli Zhao; Brian Fulkerson; Riza Alp Guler
- Web page: https://patentgrep.com/patent/US20250095308A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/60: Analysis of geometric attributes
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 15/04: Texture mapping
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2219/2016: Rotation, translation, scaling
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; determining UV positions of the real-world object depicted in the video to obtain pixel values associated with the UV positions; generating an external mesh and associated augmented reality (AR) element representing the real-world object based on the pixel values associated with the UV positions; deforming the external mesh based on changes to the 3D body mesh and a deformation parameter; and modifying the video to replace the real-world object with the AR element based on the deformed external mesh.

## Claims

### Claim 1

A method comprising:
generating, by one or more processors, a three-dimensional (3D) mesh associated with an object;
generating an external mesh for the object;
deforming the external mesh based on changes to the 3D mesh and a deformation parameter; and
modifying an image to overlay an augmented reality (AR) element over at least a portion of a depiction of the object based on the deformed external mesh.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
receiving a video that includes a depiction of a real-world object comprising the object.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining UV positions of the object depicted in the video to obtain pixel values associated with the UV positions.

### Claim 4 (depends on 1)

The method of claim 1, wherein the object comprises a person, and wherein the AR element comprises a texture that represents visual attributes of a body of the person, and wherein the 3D mesh tracks movement of the object across frames.

### Claim 5 (depends on 1)

The method of claim 1, further comprising automatically establishing a correspondence between the 3D mesh associated with the object and the external mesh, wherein the correspondence comprises placing the external mesh at a same position and orientation as the 3D mesh.

### Claim 6 (depends on 1)

The method of claim 1, wherein a video is modified in real time to depict the object being changed into the AR element while the object moves.

### Claim 7 (depends on 1)

The method of claim 1, wherein the at least the portion of the object is replaced with the AR element according to a blending parameter that comprises a linear function or a non-linear function.

### Claim 8 (depends on 1)

The method of claim 1, further comprising obtaining placement information for the external mesh that describes where to position the external mesh relative to the 3D mesh.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
computing a change in body shape, body state, or body properties of the object across frames; and
adjusting a shape of the external mesh based on the change in the shape, body state, or body properties.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
computing a change in position or scale of the object across frames; and
adjusting a position or scale of the external mesh based on the change in the position or scale.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
computing a change in rotation of the object across frames; and
rotating the external mesh in 3D based on the change in the rotation.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
determining a first set of coordinates of the 3D mesh in a normal-tangent space for a first frame of frames;
determining a second set of coordinates of the 3D mesh in the normal-tangent space for a second frame of the frames;
computing, in real time, a change between the first and second sets of coordinates in the normal-tangent space; and
transferring the change between the first and second sets of coordinates in the normal-tangent space to the external mesh.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
receiving, by an application, placement information for the external mesh from a third-party designer of the external mesh; and
adding the external mesh to the 3D mesh based on the placement information.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
generating a two-dimensional (2D) texture map of a frame;
identifying a portion of the 2D texture map corresponding to the object; and
determining UV positions of the object depicted to obtain pixel values associated with the UV positions based on the portion of the 2D texture map corresponding to the object.

### Claim 15 (depends on 1)

The method of claim 1, wherein the deformation parameter identifies a portion of the external mesh and includes a modification to the portion of the external mesh.

### Claim 16 (depends on 14)

The method of claim 14, comprising enlarging a size of the portion of the external mesh to generate an enlarged portion of the external mesh.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
determining pixel values of the AR element corresponding to an edge of the portion of the external mesh being enlarged; and
blending the pixel values associated with the AR element from the edge of the portion of the external mesh into a portion of the AR element corresponding to the enlarged portion of the external mesh.

### Claim 18 (depends on 1)

The method of claim 1, comprising reducing a size of a portion of the external mesh.

### Claim 19

A system comprising:
at least one processor of a device; and
a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating a three-dimensional (3D) mesh associated with an object;
generating an external mesh for the object;
deforming the external mesh based on changes to the 3D mesh and a deformation parameter; and
modifying an image to overlay an augmented reality (AR) element over at least a portion of a depiction of the object based on the deformed external mesh.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor of a device, cause the at least one processor to perform operations comprising:
generating a three-dimensional (3D) mesh associated with an object;
generating an external mesh for the object;
deforming the external mesh based on changes to the 3D mesh and a deformation parameter; and
modifying an image to overlay an augmented reality (AR) element over at least a portion of a depiction of the object based on the deformed external mesh.
