# US 2017/0287218 A1 · Virtual object manipulation within physical environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287218 A1
- Document ID: US20170287218A1
- Status: Published application
- Kind: A1
- Application number: 15/085,905
- Filed: 2016-03-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Benjamin Nuernberger; Hrvoje Benko; Andrew Wilson; Eyal Ofek
- Web page: https://patentgrep.com/patent/US20170287218A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2200/04: involving 3D image data
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method, computing device and head-mounted display device for manipulating a virtual object displayed via a display device are disclosed. In one example, image data of a physical environment comprising physical features is received. A three dimensional model of at least a portion of the environment is generated. Candidate anchor features that each correspond to one of the physical features are extracted from the image data. User input is received that manipulates the virtual object as displayed within the environment. Based on the manipulation, a correspondence between a virtual anchor feature of the virtual object and a corresponding candidate anchor feature is identified. An indication of the corresponding candidate anchor feature at its corresponding physical feature within the environment is displayed to the user.

## Claims

### Claim 1

A method for manipulating a virtual object displayed via a display device, the method comprising:
receiving image data of a real world physical environment comprising a plurality of physical features;
using the image data, generating a three dimensional model of at least a portion of the real world physical environment;
extracting from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment;
receiving user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;
based on the manipulation of the virtual object, identifying at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features; and
based on the manipulation of the virtual object and the identification of the at least one correspondence, displaying via the display device an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
based on receiving a user selection input, aligning the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature; and
displaying the virtual object with its virtual anchor feature aligned with the corresponding candidate anchor feature.

### Claim 3 (depends on 2)

The method of claim 2, wherein the corresponding physical feature is found on a physical object in the real world physical environment, the method further comprising, when the physical object moves within the real world physical environment, displaying a corresponding movement of the virtual object such that its virtual anchor feature remains aligned with the corresponding candidate anchor feature of the corresponding physical feature on the physical object.

### Claim 4 (depends on 2)

The method of claim 2, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the virtual object is displayed located at a virtual object location in the real world physical environment, the method further comprising, when the physical object moves within the real world physical environment, maintaining the display of the virtual object at the virtual object location.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of physical features comprises a three dimensional linear edge and a three dimensional planar surface.

### Claim 6 (depends on 1)

The method of claim 1, wherein the corresponding physical feature is located at a non-visible location in the real world physical environment that is outside a field of view of the display device, the method further comprising displaying the indication extending into the field of view of the display device from the non-visible location.

### Claim 7 (depends on 6)

The method of claim 6, further comprising, when the virtual anchor feature of the virtual object aligns with the corresponding candidate anchor feature of the corresponding physical feature located at the non-visible location, outputting a directional notification that indicates the non-visible location of the corresponding physical feature.

### Claim 8 (depends on 1)

The method of claim 1, wherein the indication of the corresponding candidate anchor feature is displayed to extend beyond the corresponding physical feature.

### Claim 9 (depends on 1)

The method of claim 1, further comprising generating a filtered plurality of candidate anchor features by filtering the plurality of candidate anchor features based on one or more of (1) a distance between the display device and each of the candidate anchor features, (2) a distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within a field of view of the display device, (4) a frequency of prior use of each of the candidate anchor features, and (5) user filtering input.

### Claim 10 (depends on 9)

The method of claim 9, wherein identifying the at least one correspondence further comprises identifying the at least one correspondence from the filtered plurality of candidate anchor features.

### Claim 11 (depends on 1)

The method of claim 1, further comprising receiving a user preselection input of a selected physical feature from the plurality of physical features, wherein the selected physical feature comprises the corresponding physical feature of the corresponding candidate anchor feature, and wherein identifying the at least one correspondence is also based on the user preselection input of the selected physical feature.

### Claim 12 (depends on 1)

The method of claim 1, wherein the corresponding candidate anchor feature is located at an initial three dimensional location in the three dimensional model, and is extracted from a frame of the image data received at an initial time, the method further comprising:
determining that the corresponding candidate anchor feature is not present in subsequent frames of the image data received after the initial time; and
maintaining the corresponding candidate anchor feature at the initial three dimensional location in the three dimensional model until expiration of a predetermined period of time after the initial time.

### Claim 13

A computing device for manipulating a virtual object displayed via a display device, the computing device comprising:
a processor; and
a memory holding instructions executable by the processor to:
receive image data of a real world physical environment comprising a plurality of physical features;
using the image data, generate a three dimensional model of at least a portion of the real world physical environment;
extract from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment;
receive user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;
based on the manipulation of the virtual object, identify at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features; and
based on the manipulation of the virtual object and the identification of the at least one correspondence, display via the display device an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the instructions are executable by the processor to:
based on receiving a user selection input, align the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature; and
display via the display device the virtual object with its virtual anchor feature aligned with the corresponding candidate anchor feature.

### Claim 15 (depends on 14)

The computing device of claim 14, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the instructions are executable by the processor to, when the physical object moves within the real world physical environment, display via the display device a corresponding movement of the virtual object such that its virtual anchor feature remains aligned with the corresponding candidate anchor feature of the corresponding physical feature on the physical object.

### Claim 16 (depends on 14)

The computing device of claim 14, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the virtual object is displayed via the display device located at a virtual object location in the real world physical environment, and the instructions are executable by the processor to, when the physical object moves within the real world physical environment, maintain the display of the virtual object at the virtual object location.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein the corresponding physical feature is located at a non-visible location in the real world physical environment that is outside a field of view of the display device, and the instructions are executable by the processor to display via the display device the indication extending into the field of view of the display device from the non-visible location.

### Claim 18 (depends on 17)

The computing device of claim 17, wherein the instructions are executable by the processor to, when the virtual anchor feature of the virtual object aligns with the corresponding candidate anchor feature of the corresponding physical feature located at the non-visible location, output a directional notification that indicates the non-visible location of the corresponding physical feature.

### Claim 19 (depends on 13)

The computing device of claim 13, wherein to identify the at least one correspondence further comprises to identify the at least one correspondence from a filtered plurality of candidate anchor features that are generated based on one or more of (1) a distance between the display device and each of the candidate anchor features, (2) a distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within a field of view of the display device, (4) a frequency of prior use of each of the candidate anchor features, and (5) user filtering input.

### Claim 20

A head-mounted display device, comprising:
an at least partially see-through display;
a processor; and
a memory holding instructions executable by the processor to:
display a virtual object via the at least partially see-through display;
receive image data of a real world physical environment comprising a plurality of physical features;
using the image data, generate a three dimensional model of at least a portion of the real world physical environment;
extract from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment;
receive user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;
based on the manipulation of the virtual object, identify at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features;
based on the manipulation of the virtual object and the identification of the at least one correspondence, display via the at least partially see-through display an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment;
based on receiving a user selection input, align the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature; and
display the virtual object with its virtual anchor feature aligned with the corresponding candidate anchor feature.