# US 12,182,951 B2 · Augmented reality content generators including 3D data in a messaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,951 B2
- Document ID: US12182951B2
- Status: Granted
- Kind: B2
- Application number: 18/240,373
- Filed: 2023-08-31
- Published: 2023-12-21
- Granted: 2024-12-31
- Earliest claimed: 2020-04-13
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Kyle Goodrich; Samuel Edward Hare; Maxim Maximov Lazarov; Tony Mathew; Andrew James McPhee; Daniel Moreno; Wentao Shang
- Web page: https://patentgrep.com/patent/US12182951B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The subject technology generates a segmentation mask based on first image data. The subject technology applies the segmentation mask on first depth data to reduce a set of artifacts in a depth map based on the first depth data. The subject technology generates a packed depth map based at least in part on the depth map. The subject technology converts a single channel floating point texture to a raw depth map. The subject technology generates multiple channels. The subject technology applies, to the first image data and the first depth data, a first augmented reality content generator corresponding to a selected first selectable graphical item, the first image data and the first depth data being captured with a camera. The subject technology generates a message including the applied first augmented reality content generator to the first image data and the first depth data.

## Claims

### Claim 1

A method, comprising:
generating a segmentation mask based on first image data;
applying the segmentation mask on first depth data to reduce a set of artifacts in a depth map based on the first depth data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map;
applying, to the first image data and the first depth data, a first augmented reality content generator corresponding to a selected first selectable graphical item, the first image data and the first depth data being captured with a camera; and
generating a message including the applied first augmented reality content generator to the first image data and the first depth data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
selecting a set of augmented reality content generators from a plurality of available augmented reality content generators, the selected set of augmented reality content generators comprising at least one augmented reality content generator for applying a three-dimensional (3D) effect; and
causing display of a carousel interface comprising a plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators.

### Claim 3 (depends on 2)

The method of claim 2, wherein the 3D effect comprises an image processing operation on the first image data or the first depth data.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
rendering a view of the first image data and the first depth data based on the applied 3D effect;
receiving movement data from a movement sensor;
updating a view of the first image data and the first depth data based at least in part on the movement data; and
rendering the updated view the first image data and the first depth data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the updated view and the view are different views of the applied 3D effect.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving a swipe gesture via a touch screen of a client device; and
in response to receiving the swipe gesture, causing navigation through a plurality of selectable graphical items.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
rendering second image data by applying the first augmented reality content generator to the first image data; and
causing display of the rendered second image data, the rendered second image data comprising an interactive 3D effect.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
performing a background in-painting technique to eliminate a portrait from a background of the first image data.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the message further includes information related to a depth map and a portrait segmentation mask.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first image data and the first depth data are captured by an optical sensor of a client device.

### Claim 11

A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
generating a segmentation mask based on first image data;
applying the segmentation mask on first depth data to reduce a set of artifacts in a depth map based on the first depth data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map;
applying, to the first image data and the first depth data, a first augmented reality content generator corresponding to a selected first selectable graphical item, the first image data and the first depth data being captured with a camera; and
generating a message including the applied first augmented reality content generator to the first image data and the first depth data.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise:
selecting a set of augmented reality content generators from a plurality of available augmented reality content generators, the selected set of augmented reality content generators comprising at least one augmented reality content generator for applying a three-dimensional (3D) effect; and
causing display of a carousel interface comprising a plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators.

### Claim 13 (depends on 12)

The system of claim 12, wherein the 3D effect comprises an image processing operation on the first image data or the first depth data.

### Claim 14 (depends on 12)

The system of claim 12, wherein the operations further comprise:
rendering a view of the first image data and the first depth data based on the applied 3D effect;
receiving movement data from a movement sensor;
updating a view of the first image data and the first depth data based at least in part on the movement data; and
rendering the updated view the first image data and the first depth data.

### Claim 15 (depends on 11)

The system of claim 11, wherein the operations further comprise:
receiving a swipe gesture via a touch screen of a client device; and
in response to receiving the swipe gesture, causing navigation through a plurality of selectable graphical items.

### Claim 16 (depends on 11)

The system of claim 11, wherein the operations further comprise:
rendering second image data by applying the first augmented reality content generator to the first image data; and
causing display of the rendered second image data, the rendered second image data comprising an interactive 3D effect.

### Claim 17 (depends on 11)

The system of claim 11, wherein the operations further comprise:
performing a background in-painting technique to eliminate a portrait from a background of the first image data.

### Claim 18 (depends on 11)

The system of claim 11, wherein generating the message further includes information related to a depth map and a portrait segmentation mask.

### Claim 19 (depends on 11)

The system of claim 11, wherein the first image data and the first depth data are captured by an optical sensor of a client device.

### Claim 20

A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
generating a segmentation mask based on first image data;
applying the segmentation mask on first depth data to reduce a set of artifacts in a depth map based on the first depth data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map;
applying, to the first image data and the first depth data, a first augmented reality content generator corresponding to a selected first selectable graphical item, the first image data and the first depth data being captured with a camera; and
generating a message including the applied first augmented reality content generator to the first image data and the first depth data.