# US 12,217,453 B2 · Mirror-based augmented reality experience

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,453 B2
- Document ID: US12217453B2
- Status: Granted
- Kind: B2
- Application number: 18/231,025
- Filed: 2023-08-07
- Published: 2023-11-23
- Granted: 2025-02-04
- Earliest claimed: 2021-10-20
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Matan Zohar; Yanli Zhao; Brian Fulkerson; Itamar Berger
- Web page: https://patentgrep.com/patent/US12217453B2

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06T 7/251: {involving models}
- G06T 7/74: {involving reference images or patches}
- G06T 7/75: {involving models}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/107: {Static hand or arm}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20101: Interactive definition of point of interest, landmark or seed
- G06T 2207/30196: Human being; Person
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person. The operations further include identifying a set of skeletal joints of the person. The operations further include identifying a pose of the person depicted in the video based on positioning of the set of skeletal joints (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device). The operations further include determining, based on the pose of the person (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device), that the video comprises a mirror reflection of the person. The operations further include, in response to determining that the video comprises the mirror reflection of the person, causing display of a 3D virtual object in the video.

## Claims

### Claim 1

A method comprising:
receiving, by one or more processors, a video that depicts a person;
determining, based on a pose of the person, that the video depicts a mirror reflection of the person and that the person is capturing the video while standing in front of a mirror; and
in response to determining that the person is capturing the video while standing in front of the mirror, causing display of a three-dimensional (3D) virtual object in the video.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
computing a 3D position for placement of the 3D virtual object relative to a 3D reference point of the person; and
updating the 3D position of the 3D virtual object in the video based on 3D movement of the person using the 3D reference point.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining that the person is capturing the video while standing in front of the mirror comprises:
obtaining a target pose representing a target user posing in front of a mirror;
comparing the pose of the person depicted in the video with the target pose; and
in response to determining that the pose of the person corresponds to the target pose, determining that the person is capturing the video while standing in front of the mirror.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
applying a neural network to the video that depicts the person to determine that the video depicts the mirror reflection, the neural network being trained to establish a relationship between a plurality of training videos depicting objects and a ground-truth mirror reflection classification.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
obtaining a set of training data comprising the plurality of training videos;
applying the neural network to a first training video of the plurality of training videos to estimate a classification that the first training video comprises a mirror reflection of a given object;
comparing the estimated classification with the ground-truth mirror reflection classification associated with the first training video; and
updating one or more parameters of the neural network based on a result of comparing the estimated classification with the ground-truth mirror reflection classification.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining that a rear-facing camera of a device is being used to capture the video, wherein determining that the video comprises the mirror reflection of the person is further performed based on determining that the rear-facing camera is being used to capture the video.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
identifying a hand of the person depicted in the video; and
determining a position of the hand of the person depicted in the video, wherein the determining that the video depicts the mirror reflection of the person is further performed based on determining the position of the hand of the person depicted in the video.

### Claim 8 (depends on 7)

The method of claim 7, further comprising determining that a pose of the hand corresponds to a target pose depicting a hand raised in front of a body of the person depicted in the video.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
determining that the hand of the person depicted in the video is holding a device, wherein the determining that the video depicts the mirror reflection of the person is further performed based on determining that the hand of the person depicted in the video is holding the device.

### Claim 10 (depends on 9)

The method of claim 9, wherein causing display of the 3D virtual object in the video comprises replacing a depiction of the device with the 3D virtual object.

### Claim 11 (depends on 10)

The method of claim 10, wherein the 3D virtual object comprises a torch or a makeup brush.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
detecting a depiction of a frame of the mirror in the video, wherein causing display of the 3D virtual object in the video comprises enlarging a size of the mirror to remove the depiction of the frame from the video, the enlarging comprising:
blending pixels inside the mirror that are within a threshold distance of the frame of the mirror with pixels of the frame of the mirror; and
blending pixels outside of the mirror that are within the threshold distance of the frame of the mirror with the pixels inside of the mirror.

### Claim 13 (depends on 12)

The method of claim 12, wherein the threshold distance is a function of a distance between an edge of the frame of the mirror and a border of a frame of the video.

### Claim 14 (depends on 1)

The method of claim 1, wherein causing display of the 3D virtual object in the video comprises at least one of retouching or brightening one or more portions of the person depicted in the video or removing a background depicted in the video.

### Claim 15 (depends on 1)

The method of claim 1, wherein the 3D virtual object is presented together with the person depicted in the video.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
determining that the person has moved from a first 3D position to a second 3D position between a first frame and a second frame of the video;
computing a distance and trajectory of movement of the person from the first 3D position to the second 3D position; and
moving the 3D virtual object from a third 3D position to a fourth 3D position based on the distance and trajectory of the movement of the person from the first 3D position to the second 3D position.

### Claim 17 (depends on 1)

The method of claim 1, wherein positioning of a set of skeletal joints of the person in the video is tracked using images captured by an RGB camera of a device without using a depth sensor.

### Claim 18

A system comprising:
one or more processors configured to perform operations comprising:
receiving a video that depicts a person;
determining, based on a pose of the person, that the video depicts a mirror reflection of the person and that the person is capturing the video while standing in front of a mirror; and
in response to determining that the person is capturing the video while standing in front of the mirror, causing display of a three-dimensional (3D) virtual object in the video.

### Claim 19 (depends on 18)

The system of claim 18, wherein positioning of a set of skeletal joints of the person in the video is tracked using images captured by an RGB camera of a device without using a depth sensor.

### Claim 20

A non-transitory machine-readable storage medium including an augmented reality system that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a video that depicts a person;
determining, based on a pose of the person, that the video depicts a mirror reflection of the person and that the person is capturing the video while standing in front of a mirror; and
in response to determining that the person is capturing the video while standing in front of the mirror, causing display of a three-dimensional (3D) virtual object in the video.