# US 2021/0233314 A1 · Systems, Methods, and Media for Automatically Triggering Real-Time Visualization of Physical Environment in Artificial Reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233314 A1
- Document ID: US20210233314A1
- Status: Published application
- Kind: A1
- Application number: 16/773,897
- Filed: 2020-01-27
- Published: 2021-07-29
- Earliest claimed: 2020-01-27
- Assignees: Facebook Technologies, LLC
- Applicants: Facebook Technologies, LLC
- Inventors: Samuel Alan Johnson; Mahdi Salmani Rahimi; Shaik Shabnam Nizamudeen Basha; Belinda Heywood
- Web page: https://patentgrep.com/patent/US20210233314A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0172: {characterised by optical features}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/50: Depth or shape recovery
- G06T 15/005: {General purpose rendering architectures}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

In particular embodiments, a computing system may display a virtual-reality scene on a device worn by a user. The system may capture an image of a physical environment surrounding the user and detect an object of interest in the image. The system may determine, based on the detected object, that one or more alert criteria are satisfied. The system may generate depth measurements of at least the object. The system may generate, based on the depth measurements, a model representing the object. The system may render, based on a viewpoint of the user and the model, an output image depicting a visualization of the object. The system may display the output image on the device worn by the user in response to the determination that the one or more alert criteria are satisfied.

## Claims

### Claim 1

A method comprising, by a computing system:
displaying a virtual-reality scene on a device worn by a user;
capturing a pair of stereo images of a physical environment surrounding the user via sensors of the device worn by the user;
detecting an object of interest in the pair of stereo images;
determining, based on the detected object, that one or more alert criteria are satisfied;
generating depth measurements of at least the object in the physical environment surrounding the user based on at least the pair of stereo images captured via the sensors of the device worn by the user;
generating, based on the depth measurements, a three-dimensional (3D) model representing the object;
rendering, based on a current viewpoint of the user in the device worn by the user and the 3D model representing the object, an output image depicting a visualization of the object in the physical environment surrounding the user; and
displaying, along with the virtual-reality scene, the output image depicting the visualization of the object in the physical environment surrounding the user according to the current viewpoint of the user on the device worn by the user in response to the determination that the one or more alert criteria are satisfied.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
capturing one or more additional images of the physical environment surrounding the user; and
detecting the object of interest in the one or more additional images;
wherein the determination that the one or more alert criteria are satisfied is further based on the object of interest being detected in the one or more additional images.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining that the object of interest in the pair of images is performing an action;
wherein the determination that the one or more alert criteria are satisfied is further based on the action performed by the object of interest.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
accessing a predefined region surrounding the user; and
predicting that the object of interest is likely to enter the predefined region;
wherein the determination that the one or more alert criteria are satisfied is further based on the prediction that the object of interest is likely to enter the predefined region.

### Claim 5 (depends on 1)

The method of claim 1, wherein the object of interest is a person.

### Claim 6

(canceled)

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The method of claim 1, wherein the device worn by the user is a head-mounted device that blocks the user from seeing the physical environment directly.

### Claim 9

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
display a virtual-reality scene on a device worn by a user;
capture a pair of stereo images of a physical environment surrounding the user via sensors of the device worn by the user;
detect an object of interest in the pair of stereo images;
determine, based on the detected object, that one or more alert criteria are satisfied;
generate depth measurements of at least the object in the physical environment surrounding the user based on at least the pair of stereo images captured via the sensors of the device worn by the user;
generate, based on the depth measurements, a three-dimensional (3D) model representing the object;
render, based on a current viewpoint of the user in the device worn by the user and the 3D model representing the object, an output image depicting a visualization of the object in the physical environment surrounding the user; and
display, along with the virtual-reality scene, the output image depicting the visualization of the object in the physical environment surrounding the user according to the current viewpoint of the user on the device worn by the user in response to the determination that the one or more alert criteria are satisfied.

### Claim 10 (depends on 9)

The media of claim 9, wherein the software is further operable when executed to:
capture one or more additional images of the physical environment surrounding the user; and
detect the object of interest in the one or more additional images;
wherein the determination that the one or more alert criteria are satisfied is further based on the object of interest being detected in the one or more additional images.

### Claim 11 (depends on 9)

The media of claim 9, wherein the software is further operable when executed to:
determine that the object of interest in the pair of images is performing an action;
wherein the determination that the one or more alert criteria are satisfied is further based on the action performed by the object of interest.

### Claim 12 (depends on 9)

The media of claim 9, wherein the software is further operable when executed to:
access a predefined region surrounding the user; and
predict that the object of interest is likely to enter the predefined region;
wherein the determination that the one or more alert criteria are satisfied is further based on the prediction that the object of interest is likely to enter the predefined region.

### Claim 13

(canceled)

### Claim 14 (depends on 9)

The media of claim 9, wherein the device worn by the user is a head-mounted device that blocks the user from seeing the physical environment directly.

### Claim 15

A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
display a virtual-reality scene on a device worn by a user;
capture a pair of stereo images of a physical environment surrounding the user via sensors of the device worn by the user;
detect an object of interest in the pair of stereo images;
determine, based on the detected object, that one or more alert criteria are satisfied;
generate depth measurements of at least the object in the physical environment surrounding the user based on at least the pair of stereo images captured via the sensors of the device worn by the user;
generate, based on the depth measurements, a three-dimensional (3D) model representing the object;
render, based on a current viewpoint of the user in the device worn by the user and the 3D model representing the object, an output image depicting a visualization of the object in the physical environment surrounding the user; and
display, along with the virtual-reality scene, the output image depicting the visualization of the object in the physical environment surrounding the user according to the current viewpoint of the user on the device worn by the user in response to the determination that the one or more alert criteria are satisfied.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more processors are further operable when executing the instructions to cause the system to:
capture one or more additional images of the physical environment surrounding the user; and
detect the object of interest in the one or more additional images;
wherein the determination that the one or more alert criteria are satisfied is further based on the object of interest being detected in the one or more additional images.

### Claim 17 (depends on 15)

The system of claim 15, wherein the one or more processors are further operable when executing the instructions to cause the system to:
determine that the object of interest in the pair of images is performing an action;
wherein the determination that the one or more alert criteria are satisfied is further based on the action performed by the object of interest.

### Claim 18 (depends on 15)

The system of claim 15, wherein the one or more processors are further operable when executing the instructions to cause the system to:
access a predefined region surrounding the user; and
predict that the object of interest is likely to enter the predefined region;
wherein the determination that the one or more alert criteria are satisfied is further based on the prediction that the object of interest is likely to enter the predefined region.

### Claim 19

(canceled)

### Claim 20 (depends on 15)

The system of claim 15, wherein the device worn by the user is a head-mounted device that blocks the user from seeing the physical environment directly.

### Claim 21 (depends on 1)

The method of claim 1, wherein detecting the object of interest in the pair of stereo images comprises:
processing the pair of stereo images using a machine-learning model to detect the object of interest.

### Claim 22 (depends on 1)

The method of claim 1, further comprising:
applying a gradient filter to the output image, wherein the gradient filter highlights edges of the detected object in the pair of stereo images, and wherein the output image after applying the gradient filter results in a line-based outline of the detected object over the virtual-reality scene.

### Claim 23 (depends on 9)

The media of claim 9, wherein to detect the object of interest in the pair of stereo images, the software is further operable when executed to:
process the pair of stereo images using a machine-learning model to detect the object of interest.

### Claim 24 (depends on 15)

The system of claim 15, wherein to detect the object of interest in the pair of stereo images, the one or more processors are further operable when executing the instructions to cause the system to:
process the pair of stereo images using a machine-learning model to detect the object of interest.
