# US 2019/0244430 A1 · Constructing augmented reality environment with pre-computed lighting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244430 A1
- Document ID: US20190244430A1
- Status: Published application
- Kind: A1
- Application number: 16/251,575
- Filed: 2019-01-18
- Published: 2019-08-08
- Earliest claimed: 2012-11-05
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jonathan Steed; Aaron Krauss; Mike Scavezze; Wei Zhang; Arthur Tomlin; Tony Ambrus; Brian Mount; Stephen Latta; Ryan Hastings
- Web page: https://patentgrep.com/patent/US20190244430A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 15/50: Lighting effects
- G06T 2215/16: Using real world measurements to influence rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments related to efficiently constructing an augmented reality environment with global illumination effects are disclosed. For example, one disclosed embodiment provides a method of displaying an augmented reality image via a display device. The method includes receiving image data, the image data capturing an image of a local environment of the display device, and identifying a physical feature of the local environment via the image data. The method further includes constructing an augmented reality image of a virtual structure for display over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect, and outputting the augmented reality image to the display device.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

Enacted via execution of instructions on a computing system comprising a processor, a method of generating a virtual image based upon image data from an image sensor, the method comprising:
identifying, via execution of the instructions, a representation of a surface of an environment in the image data;
constructing, via execution of the instructions, an image of a virtual structure for display in spatial registration with the representation of the surface by fitting one or more modular image segments of the virtual structure to the representation of the surface based on a geometry of the representation of the surface by one or more of scaling, rotating, clipping, and morphing the one or more modular image segments, each modular image segment having an appearance of at least a portion of the virtual structure; and
outputting, via execution of the instructions, the image for display.

### Claim 22 (depends on 21)

The method of claim 21, wherein the representation of the surface comprises a mesh representation of the surface.

### Claim 23 (depends on 21)

The method of claim 21, wherein fitting the one or more modular image segments of the virtual structure to the representation of the surface comprises arranging two or more modular image segments adjacent to one another.

### Claim 24 (depends on 21)

The method of claim 21, wherein fitting the one or more modular image segments of the virtual structure to the representation of the surface comprises fitting a second modular image segment to a first modular image segment based on a connectivity constraint restricting a set of other modular segments joinable to the first modular image segment.

### Claim 25 (depends on 21)

The method of claim 21, wherein at least one modular image segment comprises a pre-computed lighting effect.

### Claim 26 (depends on 25)

The method of claim 25, wherein the pre-computed lighting effect comprises a directional lighting effect.

### Claim 27 (depends on 25)

The method of claim 25, further comprising detecting a lighting characteristic of the environment, wherein the pre-computed lighting effect is selected based upon a similarity to the lighting characteristic.

### Claim 28 (depends on 21)

The method of claim 21, further comprising detecting a lighting characteristic of the environment, and modulating an appearance of the virtual structure based on the lighting characteristic.

### Claim 29 (depends on 28)

The method of claim 28, wherein the lighting characteristic comprises a color characteristic of the environment, and wherein modulating the appearance of the virtual structure includes imparting the color characteristic to the virtual structure.

### Claim 30 (depends on 28)

The method of claim 28, wherein the lighting characteristic comprises a location of a physical light in the environment, and wherein modulating the appearance of the virtual structure includes computing a lighting effect arising from a virtual point light at the location of the physical light.

### Claim 31 (depends on 21)

The method of claim 21, further comprising applying a dynamic lighting effect to the virtual structure.

### Claim 32

A computing system, comprising:
a logic subsystem comprising a processor; and
a storage subsystem comprising instructions executable by the logic subsystem to
identify, in image data, a representation of a surface of an environment;
construct an image of a virtual structure for display in spatial registration with the representation of the surface by fitting one or more modular image segments of the virtual structure to the representation of the surface based on a geometry of the representation of the surface by one or more of scaling, rotating, clipping, and morphing the one or more modular image segments, each modular image segment having an appearance of at least a portion of the virtual structure; and
output the image for display.

### Claim 33 (depends on 32)

The computing system of claim 32, wherein the instructions executable to fit the one or more modular image segments of the virtual structure to the representation of the surface are further executable to arrange two or more modular image segments adjacent to one another.

### Claim 34 (depends on 32)

The computing system of claim 32, wherein the instructions executable to fit the one or more modular image segments of the virtual structure to the representation of the surface are further executable to fit a second modular image segment to a first modular image segment based on a connectivity constraint restricting a set of other modular segments joinable to the first modular image segment.

### Claim 35 (depends on 32)

The computing system of claim 32, wherein at least one modular image segment comprises a pre-computed lighting effect.

### Claim 36 (depends on 32)

The computing system of claim 32, further comprising instructions executable to detect a lighting characteristic of the environment, and to modulate an appearance of the virtual structure based on the lighting characteristic.

### Claim 37 (depends on 36)

The computing system of claim 36, wherein the lighting characteristic comprises a location of a physical light in the environment, and wherein the instructions executable to modulate the appearance of the virtual structure are further executable to compute a lighting effect arising from a virtual point light at the location of the physical light.

### Claim 38 (depends on 32)

The computing system of claim 32, further comprising instructions executable to apply a dynamic lighting effect to the virtual structure.

### Claim 39

Enacted via execution of instructions on a computing system comprising a processor, a method of generating a virtual image based upon image data from an image sensor, the method comprising:
identifying via execution the instructions, a representation of a surface of an environment in the image data;
constructing, via execution of the instructions, an image of a virtual structure for display in spatial registration with the representation of the surface by fitting one or more modular image segments of the virtual structure to the representation of the surface based on a geometry of the representation of the surface by one or more of scaling, rotating, clipping, and morphing the one or more modular image segments, each modular image segment having an appearance of at least a portion of the virtual structure and at least one modular image segment comprising a pre-computed lighting effect;
applying, via execution of the instructions, a dynamic lighting effect to the virtual structure; and
outputting, via execution of the instructions, the image for display.

### Claim 40 (depends on 39)

The method of claim 39, wherein the dynamic lighting effect comprises a lighting effect arising from a dynamic virtual object.
