# US 2019/0378334 A1 · Augmented reality portal-based applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378334 A1
- Document ID: US20190378334A1
- Status: Published application
- Kind: A1
- Application number: 16/386,217
- Filed: 2019-04-16
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: Vulcan Inc.
- Applicants: Vulcan Inc.
- Inventors: Richard Earl Simpkinson; Richard Ignacio Zaragoza; Keith Rosema; Alan Caplan; Paul G. Allen
- Web page: https://patentgrep.com/patent/US20190378334A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537)
- G06F 16/9537: Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04W 4/029: Location-based management or tracking services
- G06T 2200/04: involving 3D image data
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

An augmented reality system identifies a region of a scene observed by an augmented reality device. The region is associated with an application executing on the device. The system receives sensor data from the device and generates an event based on the sensor data. The system determines that the event is relevant to the application, and sends the event to the augmented reality device for processing by the application. The application generates a user interface element by specifying coordinates relative to the identified region. The coordinates are translated to a location in the scene.

## Claims

### Claim 1

A system comprising:
at least one processor; and
a memory having stored thereon computer-executable instructions that, when executed by the at least one processor, cause the system to:
identify a region of a scene observed by an augmented reality device;
associate an application with the region;
receive sensor data from the augmented reality device;
generate an event based at least in part on the sensor data; and
provide the event to the application, wherein the application defines a user interface element using coordinates indicated relative to the region, and wherein the augmented reality device displays the user interface element based at least in part on translation of the coordinates to a location in the scene.

### Claim 2 (depends on 1)

The system of claim 1, wherein the application indicates the coordinates irrespective of changes to at least one of a position or orientation of the augmented reality device.

### Claim 3 (depends on 1)

The system of claim 1, wherein the region is associated with an object and the application indicates the coordinates irrespective of changes to a position of the object.

### Claim 4 (depends on 1)

The system of claim 1, wherein the memory has stored thereon computer-executable instructions that, when executed by the at least one processor, cause the system to:
update the region based at least in part on a change to at least one of a position or orientation of the augmented reality device.

### Claim 5 (depends on 1)

The system of claim 1, wherein the application defines the user interface element by at least specifying a gravitational point associated with a user interface element.

### Claim 6 (depends on 1)

The system of claim 1, wherein the memory has stored thereon computer-executable instructions that, when executed by the at least one processor, cause the system to:
estimate relevance of an event to the application, the estimate based at least in part on the position of the region relative to the event; and
determine to provide the event to the application, based at least in part on the estimated relevance.

### Claim 7 (depends on 1)

The system of claim 1, wherein the memory has stored thereon computer-executable instructions that, when executed by the at least one processor, cause the system to:
store information indicative of a subscription by the application to events of an event type; and
provide the event to the application based at least in part on determining that the event occurred proximate to the region.

### Claim 8 (depends on 1)

The system of claim 1, wherein the augmented reality device filters sensor data based at least in part on the position of the region.

### Claim 9

A computer-implemented method, comprising:
identifying a region of an augmented reality scene;
associating an application with the region;
receiving sensor data from the augmented reality device; and
providing an event to the application, the event based at least in part on the sensor data, wherein the application, in response to the event, generates a user interface element using coordinates indicated relative to the region, and wherein the augmented reality device displays the user interface based at least on translating the coordinates to a position in the augmented reality scene.

### Claim 10 (depends on 9)

The method of claim 9, wherein the coordinates are independent of changes to at least one of a position or orientation of the augmented reality device.

### Claim 11 (depends on 9)

The method of claim 9, wherein the region is associated with an object observed in the augmented reality scene and the coordinates are independent of changes to a position of the object.

### Claim 12 (depends on 11)

The method of claim 11, further comprising updating the position of the region based on a change to at least one of a position or orientation of the augmented reality device.

### Claim 13 (depends on 11)

The method of claim 11, further comprising updating the position of the region to encompass the object in response to movement of the object.

### Claim 14 (depends on 9)

The method of claim 9, further comprising estimating the relevance of an event based at least in part on proximity of the event to the region.

### Claim 15 (depends on 9)

The method of claim 9, wherein the application indicates the coordinates by at least indicating a gravitational point in the region.

### Claim 16

A non-transitory computer-readable storage medium having stored thereon executable instructions that, in response to being executed by one or more processors of an augmented reality device, cause the augmented reality device to at least:
send a request to associate an application with a region encompassing an object identified in an augmented reality scene;
send sensor data to a remote service, the sensor data selected for sending based at least in part on a location of a region encompassing an object identified in an augmented reality scene;
receive event data based at least in part on the sensor data and a position of the region in the augmented reality scene;
define a user interface element using coordinates indicated relative to the region; and
display the user interface element based at least in part on a translation of the coordinates to a position in the augmented reality scene.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the application updates the user interface element in response to an additional event, wherein the update is based at least in part on coordinates indicated independently of motion of at least one of the augmented reality device or the object.

### Claim 18 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the application indicates the coordinates of the user interface elements by at least indicating a gravitational point in the region.

### Claim 19 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, having stored thereon further executable instructions that, in response to being executed by one or more processors of an augmented reality device, cause the augmented reality device to at least:
filter sensor data based at least in part on the position of the region.

### Claim 20 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, having stored thereon further executable instructions that, in response to being executed by one or more processors of an augmented reality device, cause the augmented reality device to at least:
track the position of the region in three-dimensional space; and
select sensor data for sending to the remote service based at least in part on the position of the region in three-dimensional space.
