# US 2023/0419497 A1 · Whole body segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419497 A1
- Document ID: US20230419497A1
- Status: Published application
- Kind: A1
- Application number: 18/243,444
- Filed: 2023-09-07
- Published: 2023-12-28
- Earliest claimed: 2021-02-24
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Gal Dudovitch; Peleg Harel; Chia-Hao Hsieh; Sergei Korolev; Ma'ayan Mishin Shuvi
- Web page: https://patentgrep.com/patent/US20230419497A1

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06T 5/70: Denoising; Smoothing
- G06T 7/149: involving deformable models, e.g. active contour models
- G06T 7/174: involving the use of two or more images
- G06T 7/194: involving foreground-background segmentation
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.

## Claims

### Claim 1

A method comprising:
accessing a video feed comprising a plurality of images received prior to an image;
predicting, by a first deep neural network based on the plurality of images received prior to the image, a segmentation of a body depicted in the image;
comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and
smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating the segmentation of the body based on the image; and
applying one or more visual effects to the image based on the smoothed segmentation.

### Claim 3 (depends on 1)

The method of claim 1, further comprising training the first deep neural network by performing operations comprising:
receiving training data comprising a plurality of training images and ground truth segmentations for each of the plurality of training images;
applying the first deep neural network to a first training image of the plurality of training images to estimate a segmentation of a given body depicted in the first training image;
computing a deviation between the estimated segmentation and the ground truth segmentation associated with the first training image; and
updating parameters of the first deep neural network based on the computed deviation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the plurality of training images comprises ground truth skeletal key points of one or more bodies depicted in the respective training images.

### Claim 5 (depends on 3)

The method of claim 3, wherein the first deep neural network estimates skeletal key points of the given body depicted in the first training image.

### Claim 6 (depends on 3)

The method of claim 3, further comprising updating parameters of the first deep neural network based on a deviation between estimated skeletal key points and ground truth skeletal key points.

### Claim 7 (depends on 3)

The method of claim 3, wherein the plurality of training images comprises a plurality of image resolutions, further comprising generating a plurality of segmentation models based on the first deep neural network, a first of the plurality of segmentation models being trained based on training images having a first of the plurality of image resolutions, a second of the plurality of segmentation models being trained based on training images having a second of the plurality of image resolutions.

### Claim 8 (depends on 3)

The method of claim 3, wherein the plurality of training images comprises a plurality of labeled and unlabeled image and video data.

### Claim 9 (depends on 3)

The method of claim 3, wherein the plurality of training images comprises e a depiction of a whole body of a particular user, an image that lacks a depiction of any user, a depiction of a plurality of users, and depictions of users at different distances from an image capture device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising training the second deep neural network by performing operations comprising:
receiving training data comprising a plurality of training videos and ground truth segmentations for each of the plurality of training videos;
applying the second deep neural network to a first training video of the plurality of training videos to predict a segmentation of the body in a frame subsequent to the first training video;
computing a deviation between the predicted segmentation of the body and the ground truth segmentation of the body depicted in the frame subsequent to the first training video; and
updating parameters of the second deep neural network based on the computed deviation.

### Claim 11 (depends on 1)

The method of claim 1, further comprising applying one or more visual effects to the image based on a segmentation border associated with the smoothed segmentation.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
determining one or more device capabilities of a device used to capture the image; and
selecting a segmentation model to generate the segmentation based on the one or more device capabilities.

### Claim 13 (depends on 1)

The method of claim 1, further comprising applying a guided filter to improve segmentation quality of portions of the smoothed segmentations that are within a specified number of pixels of edges of the smoothed segmentation.

### Claim 14 (depends on 1)

The method of claim 1, further comprising replacing a background of the image with a different background or replacing portions of body depicted in the image with different visual elements.

### Claim 15

A system comprising:
at least one processor; and
a memory component having instructions stored thereon, when executed by the at least one processor, causes the at least one processor to perform operations comprising:
accessing a video feed comprising a plurality of images received prior to an image;
predicting, by a first deep neural network based on the plurality of images received prior to the image, a segmentation of a body depicted in the image;
comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and
smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.

### Claim 16 (depends on 15)

The system of claim 15, the operations further comprising:
generating the segmentation of the body based on the image; and
applying one or more visual effects to the image based on the smoothed segmentation.

### Claim 17 (depends on 15)

The system of claim 15, the operations further comprising training the first deep neural network by performing operations comprising:
receiving training data comprising a plurality of training images and ground truth segmentations for each of the plurality of training images;
applying the first deep neural network to a first training image of the plurality of training images to estimate a segmentation of a given body depicted in the first training image;
computing a deviation between the estimated segmentation and the ground truth segmentation associated with the first training image; and
updating parameters of the first deep neural network based on the computed deviation.

### Claim 18 (depends on 17)

The system of claim 17, wherein the plurality of training images comprises ground truth skeletal key points of one or more bodies depicted in the respective training images.

### Claim 19 (depends on 17)

The system of claim 17, wherein the first deep neural network estimates skeletal key points of the given body depicted in the first training image.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon, instructions when executed by at least one processor, causes the at least one processor to perform operations comprising:
accessing a video feed comprising a plurality of images received prior to an image;
predicting, by a first deep neural network based on the plurality of images received prior to the image, a segmentation of a body depicted in the image;
comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and
smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.
