# US 11,537,351 B2 · Systems and methods for virtual and augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,537,351 B2
- Document ID: US11537351B2
- Status: Granted
- Kind: B2
- Application number: 16/990,161
- Filed: 2020-08-11
- Published: 2021-02-18
- Granted: 2022-12-27
- Earliest claimed: 2019-08-12
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Rodolpho C. Cardenuto; David Charles Lundmark
- Web page: https://patentgrep.com/patent/US11537351B2

## CPC

- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G02B 27/017: {Head mounted}
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G09G 2354/00: Aspects of interface with display user
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A computer implemented method of facilitating communication between first and second users includes displaying, by a first head-worn device, a first virtual object to the first user first user wearing the first head-worn device. The method also includes displaying, by a second head-worn device, a second virtual object to the second user wearing the second head-worn device. The method further includes facilitating, by the first and second head-worn devices, communications between the first and second users using the first and second virtual objects to simulate the first and second users being present in a common environment.

## Claims

### Claim 1

A computer implemented method of sharing a virtual experience involving a real-world object and a virtual object, the method comprising:
capturing an image of a real-world object from a first user's view of the real-world object by an environmental sensor disposed on a first head-worn device worn by a first user;
generating a view of a virtual object corresponding to the real-world object from the first user's view of the real-world object, based on the captured image;
displaying, by the first head-worn device, the view of the virtual object from the first user's view of the real-world object to the first user wearing the first head-worn device; and
displaying, by a second head-worn device, the view of the virtual object from the first user's view of the real-world object to the second user wearing the second head-worn device; and
wherein the view of the virtual object to from the first user's view of the real-world object is displayed on both the first head-worn device and second head worn device in real-time with the capturing the image of a real-world object from the first user's view of the real-world object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first user is located at a first geographic location, and
wherein the second user is located in a second geographic location different from the first geographic location.

### Claim 3 (depends on 2)

The method of claim 2, further comprising modifying, by the first user, the virtual object.

### Claim 4 (depends on 2)

The method of claim 2, further comprising modifying, by the second user, the virtual object.

### Claim 5 (depends on 2)

The method of claim 2, further comprising communicating, by the first and second head-worn devices, modifications to the virtual object between the first and second head-worn devices.

### Claim 6 (depends on 2)

The method of claim 2, wherein the first and second head-worn devices are communicatively coupled by a network.

### Claim 7 (depends on 2)

The method of claim 2, further comprising providing, by the first user via a first user interface device operatively coupled to the first head-worn device, input relating to the virtual object.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first user interface device is selected from the group consisting of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device.

### Claim 9 (depends on 2)

The method of claim 2, further comprising providing, by the second user via a second user interface device operatively coupled to the second head-worn device, input relating to the virtual object.

### Claim 10 (depends on 9)

The method of claim 9, wherein the second user interface device is selected from the group consisting of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device.

### Claim 11 (depends on 2)

The method of claim 2, further comprising providing, by a sensor operatively coupled to the first and second head-worn device, input relating to the virtual object.

### Claim 12 (depends on 11)

The method of claim 11, wherein the sensor is selected from the group consisting of a camera and an optical detection device.

### Claim 13 (depends on 11)

The method of claim 11, wherein the input is selected from the group consisting of pupil size, pupil position, pupil angle, eye spherocity, eye shape, figure motion, and anatomic data relating to a user.

### Claim 14 (depends on 11)

The method of claim 11, further comprising determining, by a processor, a user's visual focal point using the input.

### Claim 15 (depends on 11)

The method of claim 11, further comprising mapping, by a processor, a portion of a user's environment using the input.

### Claim 16 (depends on 2)

The method of claim 2, further comprising providing, by an environmental sensor operatively coupled to the first and second head-worn devices, environmental input relating to a geographical location for capturing the image of the real-world object.

### Claim 17 (depends on 16)

The method of claim 16, wherein the environmental sensor is selected from the group consisting of a camera, an optical detection device, a motion capture device, a depth measurement device, a facial recognition device, an object recognition device, an audio recognition device, a CMOS sensor, a field-of-view sensor, and an optical-enhancing sensor.

### Claim 18 (depends on 16)

The method of claim 16, wherein the environmental input is selected from the group consisting of motion capture data, gesture recognition date, depth sensing data, facial recognition data, object recognition data, unique object feature recognition data, audio recognition data, voice recognition data, acoustic source localization data, noise reduction data, infrared light data, laser light data, CMOS sensor data, field-of-view data, and optical-enhancing data.

### Claim 19 (depends on 2)

The method of claim 2, wherein the virtual object is a building, and wherein the first and second head-worn devices are configured to display a virtual tour of the building to the first and second users, respectively.

### Claim 20 (depends on 2)

The method of claim 2, wherein the virtual object is a three dimensional drawing of a manufactured object.

### Claim 21 (depends on 1)

The method of claim 1, wherein a view of the virtual object displayed to the second user through the second head-worn device allows the second user to visualize activity of the first user as though the second user is looking over a shoulder of the first user.

### Claim 22 (depends on 1)

The method of claim 1, wherein the first user, through the first head-worn device, and the second user, through the second head-worn device, collaboratively view the virtual object.

### Claim 23 (depends on 1)

The method of claim 1, wherein the first user, through the first head-worn device, and the second user, through the second head-worn device, collaboratively manipulate the virtual object.

### Claim 24 (depends on 1)

The method of claim 1, wherein
the view of the virtual object displayed to the second user through the second head-worn device allows the second user to visualize activity of the first user as though the second user is looking over a shoulder of the first user, and
the first user, through the first head-worn device, and the second user, through the second head-worn device, collaboratively view and manipulate the virtual object.

### Claim 25 (depends on 24)

The method of claim 24, further comprising communicating, between the first head-worn device and the second head-worn device, media including audio, texture, vibrations, and smell detected by respective sensors of respective first and second head-worn devices.

### Claim 26 (depends on 1)

The method of claim 1, wherein the first user is located at a first geographic location and the second user is located in a second geographic location different from the first geographic location, further comprising:
modifying the virtual object based on the first user manipulating a first user interface of the first head-worn device, the first user interface selected from the group consisting of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device;
modifying the virtual object based on the second user manipulating a second user interface of the second head-worn device, the second user interface selected from the group consisting of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device,
communicating respective modifications to the virtual object between the first head-worn device and the second head-worn device through a network, wherein
the view of the virtual object displayed to the second user through the second head-worn device allows the second user to visualize activity of the first user as though the second user is looking over a shoulder of the first user, and
the first user, through the first head-worn device, and the second user, through the second head-worn device, collaboratively view and manipulate the virtual object.
