# US 2020/0402319 A1 · Cooperative augmented reality map interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402319 A1
- Document ID: US20200402319A1
- Status: Published application
- Kind: A1
- Application number: 17/013,344
- Filed: 2020-09-04
- Published: 2020-12-24
- Earliest claimed: 2017-09-29
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Nathan L. Fillhardt; Syed Mohsin Hasan; Adrian P. Lindberg
- Web page: https://patentgrep.com/patent/US20200402319A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 16/29: Geographical information databases
- G06T 7/74: {involving reference images or patches}
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 17/05: Geographic models
- G06T 19/003: {Navigation within 3D models or images}
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04W 4/02: Services making use of location information
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/30244: Camera pose
- G06T 2219/024: Multi-user, collaborative environment
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

To reduce this amount of bandwidth needed to share 3D map images between mobile devices, according to some embodiments, a user's mobile device (i.e., a host device) may identify its origin in a 3D map and a current virtual camera position relative to the origin based on the physical position of the mobile device. The mobile device may send both the origin and the virtual camera position to another mobile device (i.e., a client device) for use in rendering a corresponding image. Separately, the client device may download the 3D map images from a server, e.g., in preparation for a meeting. In this manner, the host device may send the origin to the client device once, as well as send a data stream of the current virtual camera position for use in accessing the corresponding 3D map images at the client device.

## Claims

### Claim 1

A method of providing a view of a three-dimensional map on a display of a client device, the method comprising performing, by the client device having a physical camera communicably coupled with the display:
providing a user interface for a user to select a discovery mode that uses the physical camera of the client device;
in response to a selection of the discovery mode via the user interface:
capturing one or more first images of a physical environment within which the client device resides using the physical camera;
determining an initial position and an initial orientation of the physical camera in the physical environment by comparing a first location and a first orientation of the physical camera relative to one or more objects of a set of map objects captured in the one or more first images;
specifying a set of physical positions of a set of three-dimensional objects of the three-dimensional map relative to the initial position based on an origin position and the set of map objects;
generating a first stream of virtual positions of a client virtual camera corresponding to the client device based on physical movement of the client device as measured by comparing a second location and a second orientation of the physical camera relative to one or more second images from the physical camera; and
rendering and displaying a series of images of the three-dimensional map using the first stream of virtual positions of the client virtual camera.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving a geographic indicator from a host device that indicates a current virtual camera position of the host device in the three-dimensional map, wherein the client device retrieves a set of map objects corresponding to a region of the three-dimensional map based on the geographic indicator.

### Claim 3 (depends on 2)

The method of claim 2, wherein the geographic indicator includes the origin position.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving, over a network, a second stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the second stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a trace of the second stream of virtual positions of the host virtual camera on the three-dimensional map.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
rendering and displaying a trace of the first stream of virtual positions of the client virtual camera.

### Claim 6 (depends on 1)

The method of claim 1, wherein the set of map objects are retrieved from a map server over a first network, wherein the origin position and the first stream of virtual positions are received at the client device over a second network, and wherein the first network and the second network are different.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving a selection of a follower mode via the user interface; and
in response to the selection of the follower mode via the user interface:
receiving, over a network, a third stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the third stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a third series of images of the three-dimensional map using the third stream of virtual positions of the host device.

### Claim 8

A mobile client device comprising:
a display;
a physical camera communicably coupled with the display;
a processor; and
a memory coupled to the processor, the memory storing instructions, which when executed by the processor, cause the mobile client device to perform operations including:
providing a user interface for a user to select a discovery mode that uses the physical camera of the mobile client device;
in response to a selection of the discovery mode via the user interface:
capturing one or more first images of a physical environment within which the mobile client device resides using the physical camera;
determining an initial position and an initial orientation of the physical camera in the physical environment by comparing a first location and a first orientation of the physical camera relative to one or more objects of a set of map objects captured in the one or more first images;
specifying a set of physical positions of a set of three-dimensional objects of a three-dimensional map relative to the initial position based on an origin position and the set of map objects;
generating a first stream of virtual positions of a client virtual camera corresponding to the mobile client device based on physical movement of the mobile client device as measured by comparing a second location and a second orientation of the physical camera relative to one or more second images from the physical camera; and
rendering and displaying a series of images of the three-dimensional map using the first stream of virtual positions of the client virtual camera.

### Claim 9 (depends on 8)

The mobile client device of claim 8, further comprising:
receiving a geographic indicator from a host device that indicates a current virtual camera position of the host device in the three-dimensional map, wherein the mobile client device retrieves the set of map objects corresponding to a region of the three-dimensional map based on the geographic indicator.

### Claim 10 (depends on 9)

The mobile client device of claim 9, wherein the geographic indicator includes the origin position.

### Claim 11 (depends on 8)

The mobile client device of claim 8, further comprising:
receiving, over a network, a second stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the first stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a trace of the second stream of virtual positions of a host virtual camera on the three-dimensional map.

### Claim 12 (depends on 11)

The mobile client device of claim 11, further comprising:
rendering and displaying a trace of the first stream of virtual positions of the client virtual camera.

### Claim 13 (depends on 8)

The mobile client device of claim 8, wherein the set of map objects are retrieved from a map server over a first network, wherein the origin position and the first stream of virtual positions are received at the mobile client device over a second network, and wherein the first network and the second network are different.

### Claim 14 (depends on 8)

The mobile client device of claim 8, further comprising:
receiving a selection of a follower mode via the user interface; and
in response to the selection of the follower mode via the user interface:
receiving, over a network, a third stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the third stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a third series of images of the three-dimensional map using the third stream of virtual positions of the host device.

### Claim 15

A non-transitory computer readable medium storing a plurality of instructions that, when executed by one or more processors of a computer device, cause the one or more processors to perform operations for providing a view of a three-dimensional map on a display of a client device having a physical camera communicably coupled with the display, the operations comprising:
providing a user interface for a user to select a discovery mode that uses the physical camera;
in response to a selection of the discovery mode via the user interface:
capturing one or more first images of a physical environment within which the client device resides using the physical camera;
determining an initial position and an initial orientation of the physical camera in the physical environment by comparing a first location and a first orientation of the physical camera relative to one or more objects of a set of map objects captured in the one or more first images;
specifying a set of physical positions of a set of three-dimensional objects of the three-dimensional map relative to the initial position based on an origin position and the set of map objects;
generating a first stream of virtual positions of a client virtual camera corresponding to the client device based on physical movement of the client device as measured by comparing a second location and a second orientation of the physical camera relative to one or more second images from the physical camera; and
rendering and displaying a series of images of the three-dimensional map using the first stream of virtual positions of the client virtual camera.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the operations further comprise:
receiving a geographic indicator from a host device that indicates a current virtual camera position of the host device in the three-dimensional map, wherein the client device retrieves the set of map objects corresponding to a region of the three-dimensional map based on the geographic indicator.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the geographic indicator includes the origin position.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the operations further comprise:
receiving, over a network, a second stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the second stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a trace of the second stream of virtual positions of the host virtual camera on the three-dimensional map.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the operations further comprise:
rendering and displaying a trace of the first stream of virtual positions of the client virtual camera.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the operations further comprise:
receiving a selection of a follower mode via the user interface; and
in response to the selection of the follower mode via the user interface:
receiving, over a network, a third stream of virtual positions of a host virtual camera corresponding to a host device in the three-dimensional map, wherein the third stream of virtual positions are relative to the origin position and are generated by physical movement of the host device; and
rendering and displaying a third series of images of the three-dimensional map using the third stream of virtual positions of the host device.
