# US 2018/0324229 A1 · Systems and methods for providing expert assistance from a remote expert to a user operating an augmented reality device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324229 A1
- Document ID: US20180324229A1
- Status: Published application
- Kind: A1
- Application number: 15/970,822
- Filed: 2018-05-03
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Applicants: Tsunami VR, Inc.
- Inventors: David ROSS; Beth BREWER; Anthony DUCA; Morgan Nicholas GEBBIE
- Web page: https://patentgrep.com/patent/US20180324229A1

## CPC

- H04L 65/4015: {where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference}
- G02B 27/017: {Head mounted}
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 3/147: using display panels
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 3/002: {to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT}
- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- G02B 2027/0141: {characterised by the informative content of the display}
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Providing expert assistance from a remote expert to a user operating an augmented reality device. Particular systems and methods receive, at a server, a remote assistance request from a first user device operated by a first user located at a first location, and establish a network connection between the first user device and a second user device operated by a second user located at a second location in response to the remote assistance request. Visual information captured by a camera of the first user device is provided to the second user device operated by the second user. Assistance content generated by the second user using the second user device is provided to the first user device for presentation of the assistance content to the first user.

## Claims

### Claim 1

A method for providing expert assistance from a remote expert to a user operating an augmented reality device, the method comprising:
receiving, at a server, a remote assistance request from a first user device operated by a first user located at a first location;
after receiving the remote assistance request, establishing a network connection between the first user device and a second user device operated by a second user located at a second location;
receiving visual information captured by a camera of the first user device operated by the first user, wherein the visual information includes an image of a physical object in view of the first user;
transmitting the visual information to the second user device operated by the second user;
receiving, from the second user device operated by the second user, assistance content generated by the second user using the second user device; and
transmitting the assistance content to the first user device for presentation of the assistance content to the first user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first user device is a head-mounted augmented reality user device with a display that is at least partially transparent.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second user device is a head-mounted augmented reality user device with a display that is at least partially transparent, or a head-mounted virtual reality device, stationary computer, or mobile computer with a non-transparent display.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first user device is a head-mounted augmented reality user device with a display that is at least partially transparent, and wherein the second user device is a head-mounted augmented reality user device with a display that is at least partially transparent.

### Claim 5 (depends on 1)

The method of claim 1, wherein the method further comprises: presenting the visual information on a display of the second user device.

### Claim 6 (depends on 5)

The method of claim 5, wherein the presented visual information includes the image of the physical object that is in view of the first user, wherein the assistance content is generated for display at one or more positions relative to particular parts of the physical object, and wherein the method comprises: presenting the assistance content on a display of the first user device to appear at the one or more positions relative to the particular parts of the physical object.

### Claim 7 (depends on 1)

The method of claim 1, wherein the method comprises: presenting the assistance content at predefined locations of a display of the first user device.

### Claim 8 (depends on 1)

The method of claim 1, wherein the assistance content includes visual content or audio content generated by the second user.

### Claim 9 (depends on 8)

The method of claim 8, wherein the assistance content includes instructions the first user must follow to complete a task in relation to the physical object.

### Claim 10 (depends on 8)

The method of claim 8, wherein the assistance content includes visual content generated by the second user, and wherein the method further comprises: presenting the visual content on a display of the first user device.

### Claim 11 (depends on 8)

The method of claim 8, wherein the assistance content includes audio content, and wherein the method further comprises: presenting the audio content using a speaker of the first user device.

### Claim 12 (depends on 1)

The method of claim 1, wherein the assistance content generated by the second user includes one or more movements or gestures the first user must make to complete a task in relation to the physical object in view of the first user, and wherein the method further comprises: presenting a visual representation of the one or more movements or gestures on a display of the first user device.

### Claim 13 (depends on 12)

The method of claim 12, wherein the one or more movements or gestures generated by the second user are captured using a camera of the second user device.

### Claim 14 (depends on 12)

The method of claim 12, wherein visual representations of the one or more movements or gestures are presented on a display of the first user device as virtual hands that perform the movements and gestures.

### Claim 15 (depends on 12)

The method of claim 12, wherein the one or more movements or gestures are captured using an inertial sensor of the second user device or a peripheral device that is connect to the second user device and controlled by the second user.

### Claim 16 (depends on 1)

The method of claim 1, wherein the method further comprises:
identifying the physical object;
selecting assistance information about the identified physical object; and
transmitting the assistance information to the first user device for presentation of the assistance information to the first user.

### Claim 17 (depends on 1)

The method of claim 1, wherein the first location and the second location are different.

### Claim 18 (depends on 1)

The method of claim 1, (i) wherein transmitting the visual information to the second user device operated by the second user includes generating a virtual environment that contains a virtual replica of the physical object, and transmitting the virtual environment and the virtual replica to the second user device, and (ii) wherein receiving assistance content generated by the second user using the second user device includes recording, as the assistance content, movement and gestures made by the second user in the virtual environment, wherein the movement and gestures are performed relative to the virtual replica to shown how to resolve an issue with the physical object.

### Claim 19 (depends on 1)

A system for providing expert assistance from a remote expert to a user operating an augmented reality device, wherein the system comprises one or more machines and one or more non-transitory machine-readable media storing instructions that are operable, when executed by the one or more machines, to cause the one or more machines to perform operations of the method of claim 1.

### Claim 20 (depends on 1)

One or more non-transitory machine-readable media embodying program instructions that, when executed by one or more machines, cause the one or more machines to implement the method of claim 1.
