# US 2021/0241536 A1 · Blending Virtual Environments with Situated Physical Reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241536 A1
- Document ID: US20210241536A1
- Status: Published application
- Kind: A1
- Application number: 17/227,078
- Filed: 2021-04-09
- Published: 2021-08-05
- Earliest claimed: 2019-04-22
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Andrew D. WILSON; Christian HOLZ; Eyal OFEK; Jeremy HARTMANN
- Web page: https://patentgrep.com/patent/US20210241536A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0172: {characterised by optical features}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G06T 2215/16: Using real world measurements to influence rendering
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Various embodiments are provided herein for tracking a user's physical environment, to facilitate on-the-fly blending of a virtual environment with detected aspects of the physical environment. Embodiments can be employed to facilitate virtual roaming by compositing virtual representations of detected physical objects into virtual environments. A computing device coupled to a HMD can select portions of a depth map generated based on the user's physical environment, to generate virtual objects that correspond to the selected portions. The computing device can composite the generated virtual objects into an existing virtual environment, such that the user can traverse the virtual environment while remaining aware of their physical environment. Among other things, the computing device can employ various blending techniques for compositing, and further provide image pass-through techniques for selective viewing of the physical environment while remaining fully-immersed in virtual reality.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 1

A computer-implemented method for blending physical reality with virtual reality, the method comprising:
receiving, by a computing device, environmental sensor data from a set of environmental sensors in a physical environment;
generating, by the computing device, a colorized geometric representation of an object in the physical environment based on the received environmental sensor data;
rendering, by the computing device, a blended virtual environment based on the generated colorized geometric representation of the object and a virtual environment stored in memory; and
providing, by the computing device, the rendered blended virtual environment for display to a head-mounted display (HMD) coupled to the computing device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the environmental sensor data includes depth map data and image data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the geometric representation of the portion of the physical environment is generated based on the depth map data, texturized based on visual data extracted from the image data, and colorized based on color information extracted from each of the image data and the stored virtual environment.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
segmenting, by the computing device, a portion of the depth map data corresponding to the object from the depth map data corresponding to the physical environment.

### Claim 5 (depends on 4)

The method of claim 4, wherein the portion of the depth map data is segmented from the depth map data based on a determination that the portion of the depth map data corresponds to a known object.

### Claim 6 (depends on 4)

The method of claim 4, wherein the object is a determined salient object.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
removing, by the computing device, segmented portions of the depth map data corresponding to non-salient objects detected in the physical environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein the colorized geometric representation is generated in response to receiving the environmental sensor data, the blended virtual environment is rendered in response to generating the colorized geometric representation, and the rendered blended virtual environment is provided for display in response to rendering the blended virtual environment.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
modifying, by the computing device, colors of the colorized geometric representation based on a hue or a color scheme of the virtual environment.

### Claim 10

A computer storage medium storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving environmental sensor data from a set of environmental sensors in a physical environment;
generating a colorized geometric representation of an object in the physical environment based on the received environmental sensor data and a color scheme of a virtual environment stored in memory;
rendering a blended virtual environment based on the generated colorized geometric representation of the object and the stored virtual environment; and
providing the rendered blended virtual environment for display to a head-mounted display (HMD) coupled to the computing device.

### Claim 11 (depends on 10)

The medium of claim 10, wherein the environmental sensor data includes depth map data and image data.

### Claim 12 (depends on 11)

The medium of claim 11, wherein the geometric representation of the portion of the physical environment is generated based on the depth map data, textured based on visual data extracted from the image data, and colorized based on color information extracted from the image data and the stored virtual environment.

### Claim 13 (depends on 11)

The medium of claim 11, the operations further comprising:
segmenting a portion of the depth map data corresponding to the object from the depth map data corresponding to the physical environment.

### Claim 14 (depends on 13)

The medium of claim 13, wherein the portion of the depth map data is segmented from the depth map data based on a determination that the portion of the depth map data corresponds to a known object.

### Claim 15 (depends on 13)

The medium of claim 13, wherein the object is a determined salient object.

### Claim 16 (depends on 15)

The medium of claim 15, the operations further comprising:
removing segmented portions of the depth map data corresponding to non-salient objects detected in the physical environment.

### Claim 17 (depends on 10)

The medium of claim 10, wherein the colorized geometric representation is generated in response to receiving the environmental sensor data, the blended virtual environment is rendered in response to generating the colorized geometric representation, and the rendered blended virtual environment is provided for display in response to rendering the blended virtual environment.

### Claim 18

A computerized system comprising:
at least one processor; and
at least one computer storage medium storing computer-useable instructions that, when used by the at least one processor, cause the at least one processor to perform operations comprising:
receiving environmental sensor data from a set of environmental sensors in a physical environment;
generating a colorized geometric representation of an object in the physical environment based on the received environmental sensor data and a color distribution of a virtual environment stored in memory;
rendering a blended virtual environment based on the generated colorized geometric representation of the object and the stored virtual environment; and
providing the rendered blended virtual environment for display to a head-mounted display (HMD) coupled to the computing device.

### Claim 19 (depends on 18)

The system of claim 18, wherein the geometric representation of the portion of the physical environment is generated based on depth map data included in the environmental sensor data and is colorized based on color information extracted from image data included in the environmental sensor data and the stored virtual environment.

### Claim 20 (depends on 18)

The system of claim 18, the operations further comprising:
segmenting a portion of the depth map data corresponding to the object from the depth map data corresponding to the physical environment.
