# US 2022/0284682 A1 · Generating 3d data in a messaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284682 A1
- Document ID: US20220284682A1
- Status: Published application
- Kind: A1
- Application number: 17/525,612
- Filed: 2021-11-12
- Published: 2022-09-08
- Earliest claimed: 2019-08-28
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Kyle Goodrich; Samuel Edward Hare; Maxim Maximov Lazarov; Tony Mathew; Andrew James McPhee; Daniel Moreno; Dhritiman Sagar; Wentao Shang
- Web page: https://patentgrep.com/patent/US20220284682A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06N 20/00: Machine learning
- G06T 7/194: involving foreground-background segmentation
- G06T 7/50: Depth or shape recovery
- G06T 7/507: from shading (G06T7/586 takes precedence)
- G06T 15/50: Lighting effects
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- G06N 3/045: Combinations of networks
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30201: Face
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2024: Style variation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The subject technology receives image data and depth data. The subject technology selects an augmented reality content generator corresponding to a three-dimensional (3D) effect. The subject technology applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator. The subject technology generates, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.

## Claims

### Claim 1

A method, comprising:
applying, using a processor, a three-dimensional (3D) effect to image data and depth data based at least in part on an augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map; and
generating, using the processor, a message including information related to the applied 3D effect, the image data, and the depth data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating a depth inpainting mask based at least in part on the depth map; and
performing inpainting of the depth map using the depth inpainting mask to generate an inpainted depth map.

### Claim 3 (depends on 2)

The method of claim 2, wherein the inpainted depth map extends a foreground region of the depth map, the extended foreground region being used to perform at least one image processing operation on a region of the image data corresponding to a representation of a portion of a face corresponding to hair, an ear, or a shoulder.

### Claim 4 (depends on 2)

The method of claim 2, wherein the inpainted depth map corresponds to a post-processed depth map, the post-processed depth map includes segmentation applied to an alpha channel.

### Claim 5 (depends on 2)

The method of claim 2, wherein performing inpainting of the depth map using the depth inpainting mask comprises:
cleaning up, using the depth inpainting mask, a set of artifacts in the depth map.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the packed depth map comprises:
converting a single channel floating point texture to a raw depth map; and
generating multiple channels based at least in part on the raw depth map.

### Claim 7 (depends on 6)

The method of claim 6, wherein the multiple channels are lower precision than the single channel floating point texture and undergo multiple image processing operation without losing additional precision.

### Claim 8 (depends on 2)

The method of claim 2, further comprising:
generating a depth normal map based at least in part on the depth map; and
providing a post-process foreground image by applying, using the depth normal map, the 3D effect to a foreground region of the image data.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
sending the message to a messaging server for storing the message, the message including a set of assets, the set of assets comprising a post-processed foreground image, a post-processed depth map, and a background inpainted image.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving a command to initiate sharing of the stored message to at least one receiving device; and
sending the stored message at least one receiving device.

### Claim 11

A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
applying, using a processor, a three-dimensional (3D) effect to image data and depth data based at least in part on an augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map; and
generating, using the processor, a message including information related to the applied 3D effect, the image data, and the depth data.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise:
generating a depth inpainting mask based at least in part on the depth map; and
performing inpainting of the depth map using the depth inpainting mask to generate an inpainted depth map.

### Claim 13 (depends on 12)

The system of claim 12, wherein the inpainted depth map extends a foreground region of the depth map, the extended foreground region being used to perform at least one image processing operation on a region of the image data corresponding to a representation of a portion of a face corresponding to hair, an ear, or a shoulder.

### Claim 14 (depends on 12)

The system of claim 12, wherein the inpainted depth map corresponds to a post-processed depth map, the post-processed depth map includes segmentation applied to an alpha channel.

### Claim 15 (depends on 12)

The system of claim 12, wherein performing inpainting of the depth map using the depth inpainting mask comprises:
cleaning up, using the depth inpainting mask, a set of artifacts in the depth map.

### Claim 16 (depends on 11)

The system of claim 11, wherein generating the packed depth map comprises:
converting a single channel floating point texture to a raw depth map; and
generating multiple channels based at least in part on the raw depth map.

### Claim 17 (depends on 16)

The system of claim 16, wherein the multiple channels are lower precision than the single channel floating point texture and undergo multiple image processing operation without losing additional precision.

### Claim 18 (depends on 12)

The system of claim 12, wherein the operations further comprise:
generating a depth normal map based at least in part on the depth map; and
providing a post-process foreground image by applying, using the depth normal map, the 3D effect to a foreground region of the image data.

### Claim 19 (depends on 11)

The system of claim 11, wherein the operations further comprise:
sending the message to a messaging server for storing the message, the message including a set of assets, the set of assets comprising a post-processed foreground image, a post-processed depth map, and a background inpainted image.

### Claim 20

A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
applying, using a processor, a three-dimensional (3D) effect to image data and depth data based at least in part on an augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map; and
generating, using the processor, a message including information related to the applied 3D effect, the image data, and the depth data.