# US 2021/0279467 A1 · Semantically tagged virtual and physical objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279467 A1
- Document ID: US20210279467A1
- Status: Published application
- Kind: A1
- Application number: 17/303,193
- Filed: 2021-05-24
- Published: 2021-09-09
- Earliest claimed: 2019-07-10
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Scott D. PETILL; Robert James JARRETT; Jacob A. BOMGARDNER
- Web page: https://patentgrep.com/patent/US20210279467A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06V 20/20: in augmented reality scenes
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 40/44: Statistical methods, e.g. probability models
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A head mounted display device is provided that includes a display device, a camera device, an input device, and a processor. The processor is configured to store a database of physical objects and virtual objects that have been associated with one or more semantic tags. The processor is further configured to receive a natural language input from a user via the input device and perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input. The processor is further configured to select a target virtual object and a target physical object based on the identified one or more semantic tags, perform the determined user specified operation on the target virtual object based on the target physical object, and display the target virtual object at a physical location associated with the target physical object.

## Claims

### Claim 1

A head mounted display device comprising:
a display device configured to display virtual objects at locations in a physical environment;
a camera device configured to capture images of the physical environment;
an input device configured to receive a user input; and
a processor configured to:
store a database of physical objects and virtual objects that have been associated with one or more semantic tags;
receive a natural language input from a user via the input device;
perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input;
select a target virtual object and a target physical object from the physical objects and virtual objects in the database based on the identified one or more semantic tags;
perform the determined user specified operation on the target virtual object based on the target physical object; and
display the target virtual object at a physical location associated with the target physical object.

### Claim 2 (depends on 1)

The head mounted display device of claim 1, wherein the physical objects are selected from the group consisting of a room, a wall, a room furnishing, a table, a chair, a surface, physical location, and a person.

### Claim 3 (depends on 1)

The head mounted display device of claim 1, wherein the physical location of the target physical object is outside of a field of view of the camera device of the head mounted display device.

### Claim 4 (depends on 1)

The head mounted display device of claim 1, wherein the natural language input is a voice input received via the input device.

### Claim 5 (depends on 1)

The head mounted display device of claim 1, wherein the determined user specified operation is a move operation, and to perform the determined user specified operation the processor is further configured to:
update a location of the target virtual object in the physical environment based on the physical location associated with the target physical object; and
display the target virtual object at the updated location via the display device.

### Claim 6 (depends on 1)

The head mounted display device of claim 1, wherein the determined user specified operation is an application start operation, and to perform the determined user specified operation the processor is further configured to:
select a target application program from a plurality of application programs executable by the processor based on the identified one or more semantic tags;
generate a target virtual object associated with the target application program; and
display the generated target virtual object at a virtual location based on the physical location associated with the target physical object.

### Claim 7 (depends on 1)

The head mounted display device of claim 1, wherein the processor is further configured to:
determine a user indicated direction for a user of the head mounted display device; and
select the target virtual object or the target physical object further based on the determined user indicated direction.

### Claim 8 (depends on 7)

The head mounted display device of claim 7, wherein the user indicated direction is determined based on a detected gaze direction of the user or a detected hand gesture of the user.

### Claim 9 (depends on 1)

The head mounted display device of claim 1, further comprising a deep neural network processor configured to:
receive scene data of the physical environment captured by the camera device;
detect a physical object in the physical environment;
recognize the physical object based on a trained artificial intelligence machine learning model; and
store a reference to the recognized physical object in the database with an associated semantic tag.

### Claim 10 (depends on 9)

The head mounted display device of claim 9, wherein the processor is further configured to:
receive a user input directed to the detected physical object in the physical environment, the user input including a user specified semantic tag; and
associate the detected physical object with the user specified semantic tag in the database.

### Claim 11 (depends on 1)

The head mounted display device of claim 1, wherein the processor is further configured to:
determine a confidence value for the selection of the target virtual object or the target physical object;
based on determining that the confidence value is below a threshold value, present a query to the user for a user confirmation of the selection of the target virtual object or the target physical object.

### Claim 12

A method comprising:
at a processor:
storing a database of physical objects in a physical environment and virtual objects that have been associated with one or more semantic tags;
receiving a natural language input from a user via an input device;
performing semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input;
selecting a target virtual object and a target physical object from the physical objects and virtual objects in the database based on the identified one or more semantic tags;
performing the determined user specified operation on the target virtual object based on the target physical object; and
displaying the target virtual object at a physical location associated with the target physical object.

### Claim 13 (depends on 12)

The method of claim 12, wherein the physical objects are selected from the group consisting of a room, a wall, a room furnishing, a table, a chair, a surface, physical location, and a person.

### Claim 14 (depends on 12)

The method of claim 12, wherein the physical location of the target physical object is outside of a field of view of a camera device of a head mounted display device that includes the processor.

### Claim 15 (depends on 12)

The method of claim 12, wherein the natural language input is a voice input received via an input device.

### Claim 16 (depends on 12)

The method of claim 12, wherein the determined user specified operation is a move operation, and performing the determined user specified operation further comprises:
updating a location of the target virtual object in the physical environment based on the physical location associated with the target physical object; and
displaying the target virtual object at the updated location via the display device.

### Claim 17 (depends on 12)

The method of claim 12, wherein the determined user specified operation is an application start operation, and performing the determined user specified operation further comprises:
selecting a target application program from a plurality of application programs executable by the processor based on the identified one or more semantic tags;
generating a target virtual object associated with the target application program; and
displaying the generated target virtual object at a virtual location based on the physical location associated with the target physical object.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
receiving scene data of the physical environment captured by a camera device;
detecting a physical object in the physical environment;
recognizing the physical object based on a trained artificial intelligence machine learning model; and
storing a reference to the recognized physical object in the database with an associated semantic tag.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
receiving a user input directed to the detected physical object in the physical environment, the user input including a user specified semantic tag; and
associating the detected physical object with the user specified semantic tag in the database.

### Claim 20

A computer device comprising:
a display device configured to display virtual objects at locations in a physical environment;
a camera device configured to capture images of the physical environment;
an input device configured to receive a user input; and
a processor configured to:
receive a natural language input from a user via the input device;
perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input;
select a target virtual object and a target physical object based on the identified one or more semantic tags;
perform the determined user specified operation on the target virtual object based on the target physical object; and
display the target virtual object via the display device.
