# US 12,112,436 B2 · Systems and methods of implementing real-world ambient variance in augmented environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,436 B2
- Document ID: US12112436B2
- Status: Granted
- Kind: B2
- Application number: 17/703,837
- Filed: 2022-03-24
- Published: 2022-09-29
- Granted: 2024-10-08
- Earliest claimed: 2021-03-24
- Assignees: Wormhole Labs, Inc.
- Applicants: Wormhole Labs, Inc.
- Inventors: Curtis Hutten; Robert D. Fish
- Web page: https://patentgrep.com/patent/US12112436B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 15/20: Perspective computation
- G10L 21/034: Automatic adjustment
- H04S 1/007: {in which the audio signals are in digital form (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 7/303: {Tracking of listener position or orientation}
- G06F 3/147: using display panels
- G06T 2200/28: involving image processing hardware
- G06T 2219/024: Multi-user, collaborative environment
- G09G 3/003: {to produce spatial visual effects}
- G09G 2360/121: using a cache memory
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F)
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04S: STEREOPHONIC SYSTEMS
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

In a computer-implemented method of implementing real-world limitations in shared environment, an environment engine is configured to receive a user selection of a shared environment, wherein the shared environment represents a real-world environment. The environment engine retrieves spatial data associated with the shared environment and retrieves environmental sensor data associated with the shared environment. The environmental sensor data is contemplated to capture a varying ambient environment of the shared environment, which includes at least one of audio data and image data associated with the real-world environment. The environment engine is further configured to render a first user avatar and a second user avatar within the shared environment.

## Claims

### Claim 1

A computer-implemented method of incorporating real-world limitations into a shared, environment, concurrently renderable by first and second users, comprising:
receiving a user selection of a shared environment, wherein the shared environment represents a real-world environment;
retrieving spatial data associated with the shared environment;
retrieving environmental sensor data associated with the shared environment, wherein the environmental sensor data captures a varying ambient environment of the shared environment, and wherein the environmental sensor data includes at least one of audio data and image data;
rendering an avatar of the first user within the shared environment from a first third-person perspective; and
rendering an avatar of the second user within the shared environment from a second third-person perspective.

### Claim 2 (depends on 1)

The method of claim 1, further comprising using at least stereo rendering to the first and second users, of sounds actually occurring within the real-world environment, such that the first and second users perceive the sounds to be positionally oriented with respect to their respective avatars.

### Claim 3 (depends on 1)

The method of claim 1, further comprising rendering to the second user, apparent speech of the first avatar using a volume that varies according to varying apparent distances of the first and second avatars, within the shared environment.

### Claim 4 (depends on 1)

The method of claim 1, further comprising rendering to the second user, apparent speech of the first avatar using a volume that varies according to an apparent facial direction of the first avatar with respect to the second avatar, within the shared environment.

### Claim 5 (depends on 1)

The method of claim 1, further comprising rendering to the second user, apparent speech of the first avatar using a volume that varies according to apparent facial directions of both the first and second avatars, within the shared environment.

### Claim 6 (depends on 1)

The method of claim 1, further comprising rendering to the second user, apparent speech of the first avatar using a volume that varies according to apparent hand movements of the first avatar, with respect to the mouth of the first avatar, within the shared environment.

### Claim 7 (depends on 1)

The method of claim 1, further comprising rendering to at least one of the first and second users, a sound corresponding to an apparent interaction of a body part of the first avatar with respect to an object in the real-world environment.

### Claim 8 (depends on 7)

The method of claim 7, wherein the apparent interaction comprises at least one of the first avatar (a) stamping its foot on the floor, hitting a wall with a hand, and using a hand to open a door.

### Claim 9 (depends on 1)

The method of claim 1, further comprising rendering to the first user, volumes of sounds occurring in the real-world that vary according to apparent distances between a source of the sounds and the first avatar, within the shared environment.

### Claim 10 (depends on 1)

The method of claim 1, further comprising rendering mouth movements of the first avatar to correspond with text created or spoken speech of the first user, in a manner that is visible to both the first and second users.

### Claim 11 (depends on 1)

The method of claim 1, further comprising rendering limb movements of the first avatar to correspond with physical limb movements of the first user, in a manner that is visible to both the first and second users.

### Claim 12 (depends on 1)

The method of claim 1, further comprising rendering walking movements of the first avatar to correspond with console-based instructions from the first user, in a manner that is visible to both the first and second users.

### Claim 13 (depends on 1)

The method of claim 1, further comprising rendering facial expressions of the first avatar to correspond with console-based instructions from the first user, in a manner that is visible to both the first and second users.

### Claim 14 (depends on 1)

The method of claim 1, wherein the first third-person perspective is different from the second third-person perspective.

### Claim 15 (depends on 1)

The method of claim 1, wherein at least one of the first and second users can control sharing of the first third-person perspective with the second person.

### Claim 16 (depends on 1)

The method of claim 1, further comprising providing the first user with a control to manipulate the first third-person perspective.