# US 12,118,785 B2 · Systems and methods for providing an immersive user interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,785 B2
- Document ID: US12118785B2
- Status: Granted
- Kind: B2
- Application number: 18/221,868
- Filed: 2023-07-13
- Published: 2023-11-09
- Granted: 2024-10-15
- Earliest claimed: 2016-12-06
- Assignees: Enviropedia, Inc.
- Applicants: Enviropedia, Inc.
- Inventors: Donn M. Gurule
- Web page: https://patentgrep.com/patent/US12118785B2

## CPC

- H04N 21/8583: {by creating hot-spots}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G06F 16/29: Geographical information databases
- G06F 16/538: Presentation of query results
- G06F 16/54: Browsing; Visualisation therefor
- G06F 16/74: Browsing; Visualisation therefor (end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic programme guides, H04N21/472)
- G06F 16/9038: Presentation of query results
- G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537)
- G06T 11/65: on geographic maps
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06V 20/20: in augmented reality scenes
- H04N 21/2187: Live feed
- H04N 21/25841: {involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4884: {for displaying subtitles}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- G01C 21/3673: {Labelling using text of road map data items, e.g. road names, POI names}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- H04N 7/183: {for receiving images from a single remote source}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

Certain aspects of the present disclosure provide techniques for providing a chronologically-based search engine. Example methods includes presenting, on a display of an end-user device, live image data being captured by a camera of the end-user device, determining a pose of the end-user device, and sending the pose of the device to a server. The method also includes receiving location specific data from the server, the location specific data based on the pose of the end-user device, the location specific data including a first topic, and superimposing, on the live image data, based on the location specific data, (i) at least one highlighted area of an object associated with the first topic, and (ii) a query panel associated with the first topic superimposed proximate the object, the at least one highlighted area and the query panel being interactive elements associated with the first topic.

## Claims

### Claim 1

A method for facilitating an immersive user experience, comprising:
displaying, on a display screen of a user device, map geometry data;
sending, to a remote data provider, pre-query data associated with the map geometry data;
superimposing, on the display screen of the user device, a semi-transparent image covering a selectable point of interest in the map geometry data based on a pre-query response received in response to the pre-query data;
receiving, via an input device of the user device, a selection corresponding to the point of interest;
sending, to the remote data provider, a query associated with the selection;
receiving, from the remote data provider, one or more query results associated with the selection; and
superimposing, on the display screen of the user device, a replacement image over the point of interest based on the one or more query results.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving a second selection of a query result of the one or more query results; and
displaying, on the display screen of the user device, information associated with the selected query result.

### Claim 3 (depends on 1)

The method of claim 1, wherein the pre-query data includes one or more of:
the map geometry data;
one or more coordinates associated with the map geometry data;
an angle associated with a viewing angle of the map geometry data;
image data displayed on the display screen of the user device;
pose data associated with the user device;
field of view data associated with a camera of the user device; or
image depth mapping data.

### Claim 4 (depends on 1)

The method of claim 1, further comprising displaying on the display screen of the user device, an inset map indicating an orientation of the map geometry data.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
determining, via at least one of an accelerometer or a magnetometer of the user device, an orientation of the user device; and
determining, based on the orientation of the user device, the orientation of the map geometry data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining, via at least one of an accelerometer or a magnetometer of the user device, a change of orientation of the user device; and
displaying, on the display screen of the user device, new map geometry data based on the change of orientation of the user device.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
displaying, on the display screen of the user device, live image data generated by a camera of the user device,
wherein the map geometry data is superimposed on the live image data.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the display screen comprise a touch screen display device, and
the input device comprises the touch screen display device.

### Claim 9 (depends on 1)

The method of claim 1, wherein the user device comprises on of a phone, a tablet, a virtual reality headset, a virtual cockpit, an optical head-mounted display, or augmented reality smart glasses.

### Claim 10 (depends on 1)

The method of claim 1, further comprising transforming the replacement image superimposed over the point of interest based on a pose of the display device.

### Claim 11

A processing device, comprising a memory storing executable instructions; and a processor communicatively coupled to the memory and configured to execute the executable instructions to cause the processing device to:
display, on a display screen of a processing device, map geometry data;
send, to a remote data provider, pre-query data associated with the map geometry data;
superimpose, on the display screen of the user device, a semi-transparent image covering a selectable point of interest in the map geometry data based on a pre-query response received in response to the pre-query data;
receive, via an input device of the processing device, a selection corresponding to the point of interest;
send, to the remote data provider, a query associated with the selection;
receive, from the remote data provider, one or more query results associated with the selection; and
superimpose, on the display screen of the user device, a replacement image over the point of interest based on the one or more query results.

### Claim 12 (depends on 11)

The processing device of claim 11, wherein the processor is further configured to cause the processing device to:
receive a second selection of a query result of the one or more query results; and
display, on the display screen of the processing device, information associated with the selected query result.

### Claim 13 (depends on 11)

The processing device of claim 11, wherein the pre-query data includes one or more of:
the map geometry data;
one or more coordinates associated with the map geometry data;
an angle associated with a viewing angle of the map geometry data;
image data displayed on the display screen of the processing device;
pose data associated with the processing device;
field of view data associated with a camera of the processing device; or
image depth mapping data.

### Claim 14 (depends on 11)

The processing device of claim 11, wherein the processor is further configured to cause the processing device to display, on the display screen of the processing device, an inset map indicating an orientation of the map geometry data.

### Claim 15 (depends on 14)

The processing device of claim 14, wherein the processor is further configured to cause the processing device to:
determine, via at least one of an accelerometer or a magnetometer of the processing device, an orientation of the processing device; and
determine, based on the orientation of the processing device, the orientation of the map geometry data.

### Claim 16 (depends on 11)

The processing device of claim 11, wherein the processor is further configured to cause the processing device to:
determine, via at least one of an accelerometer or a magnetometer of the processing device, a change of orientation of the processing device; and
display, on the display screen of the processing device, new map geometry data based on the change of orientation of the processing device.

### Claim 17 (depends on 11)

The processing device of claim 11, wherein the processor is further configured to cause the processing device to:
display, on the display screen of the processing device, live image data generated by a camera of the processing device,
wherein the map geometry data is superimposed on the live image data.

### Claim 18 (depends on 11)

The processing device of claim 11, wherein:
the display screen comprise a touch screen display device, and
the input device comprises the touch screen display device.

### Claim 19 (depends on 11)

The processing device of claim 11, wherein the processing device comprises on of a phone, a tablet, a virtual reality headset, a virtual cockpit, an optical head-mounted display, or augmented reality smart glasses.