# US 2024/0004464 A1 · Transmodal input fusion for multi-user group intent processing in virtual environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004464 A1
- Document ID: US20240004464A1
- Status: Published application
- Kind: A1
- Application number: 18/252,574
- Filed: 2021-11-09
- Published: 2024-01-04
- Earliest claimed: 2020-11-13
- Applicants: Magic Leap, Inc.
- Inventors: Paul Lacey; Brian David Schwab; Samuel A. Miller; John Andrew Sands; Colman Thomas Bryant
- Web page: https://patentgrep.com/patent/US20240004464A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 15/20: Perspective computation
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This document describes imaging and visualization systems in which the intent of a group of users in a shared space is determined and acted upon. In one aspect, a method includes identifying, for a group of users in a shared virtual space, a respective objective for each of two or more of the users in the group of users. For each of the two or more users, a determination is made, based on inputs from multiple sensors having different input modalities, a respective intent of the user. At least a portion of the multiple sensors are sensors of a device of the user that enables the user to participate in the shared virtual space. A determination is made, based on the respective intent, whether the user is performing the respective objective for the user. Output data is generated and provided based on the respective objectives respective intents.

## Claims

### Claim 1

A method performed by one or more data processing apparatus, the method comprising:
identifying a respective objective for each of two or more users in a group of users;
for each of two or more users:
determining, based on inputs from multiple sensors having different input modalities, a respective intent of the user, wherein at least a portion of the multiple sensors are sensors of a device of the user that enables the user to participate in a shared virtual space; and
determining, based on the respective intent of the user, whether the user is performing a respective objective for the user;
generating, for the group of users, output data based on a respective objective for each of the two or more users and a respective intent for each of the two or more users; and
providing, to a respective device of each of one or more users in the group of users, the output data for presentation at the respective device of each of the one or more users.

### Claim 2 (depends on 1)

The method of claim 1, wherein a respective objective for at least one user of two or more users comprises at least one of (i) a task to be performed by at least one user or (ii) a subject object to which at least one user should be looking.

### Claim 3 (depends on 2)

The method of claim 2, wherein identifying a respective objective for each of two or more users, comprises determining, as the subject object, a target to which at least a threshold amount of users in the group of users is looking.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
a respective device of each user comprises a wearable device; and
determining a respective intent of a user comprises:
receiving, from the wearable device of the user, gaze data specifying a gaze of the user, gesture data specifying a hand gesture of the user, and direction data specifying a direction in which the user is moving; and
determining, as the respective intent of the user, an intent of the user with respective to a target object based on the gaze data, the gesture data, and the direction data.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
generating, for the group of users, output data based on a respective objective for each of two or more users and a respective intent for each of two or more users, comprises determining that a particular user is not performing the respective objective for the particular user; and
providing, to a respective device of each of one or more users in the group of users, the output data for presentation at the respective device of each of one or more users, comprises providing, to a device of a leader user, data indicating the particular user and data indicating that the particular user is not performing the respective objective for the particular user.

### Claim 6 (depends on 1)

The method of claim 1, wherein the output data comprises a heat map indicating an amount of users in the group of users performing a respective objective of a user.

### Claim 7 (depends on 1)

The method of claim 1, further comprising performing an action based on the output data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the action comprises reassigning one or more users to a different objective based on the output data.

### Claim 9

A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform operations comprising:
identifying, for a group of users in a shared virtual space, a respective objective for each of two or more users in the group of users;
for each of two or more users:
determining, based on inputs from multiple sensors having different input modalities, a respective intent of the user, wherein at least a portion of the multiple sensors are sensors of a device of the user that enables the user to participate in the shared virtual space; and
determining, based on the respective intent of the user, whether the user is performing a respective objective for the user;
generating, for the group of users, output data based on a respective objective for each of the two or more users and a respective intent for each of the two or more users; and
providing, to a respective device of each of one or more users in the group of users, the output data for presentation at the respective device of each of the one or more users.

### Claim 10 (depends on 9)

The computer-implemented system of claim 9, wherein a respective objective for at least one user of two or more users, comprises at least one of (i) a task to be performed by at least one user or (ii) a subject object to which at least one user should be looking.

### Claim 11 (depends on 10)

The computer-implemented system of claim 10, wherein identifying a respective objective for each of the two or more users comprises determining, as the subject object, a target to which at least a threshold amount of users in the group of users is looking.

### Claim 12 (depends on 9)

The computer-implemented system of claim 9, wherein:
a respective device of each user comprises a wearable device; and
determining a respective intent of a user comprises:
receiving, from the wearable device of the user, gaze data specifying a gaze of the user, gesture data specifying a hand gesture of the user, and direction data specifying a direction in which the user is moving; and
determining, as the respective intent of the user, an intent of the user with respective to a target object based on the gaze data, the gesture data, and the direction data.

### Claim 13 (depends on 9)

The computer-implemented system of claim 9, wherein:
generating the output data based on the respective objective for each of the two or more users and the respective intent for each of the two or more users comprises determining that a particular user is not performing the respective objective for the particular user; and
providing, to a respective device of each of one or more users in the group of users, the output data for presentation at the respective device of each of the one or more users comprises providing, to a device of a leader user, data indicating the particular user and data indicating that the particular user is not performing the respective objective for the particular user.

### Claim 14 (depends on 9)

The computer-implemented system of claim 9, wherein the output data comprises a heat map indicating an amount of users in the group of users performing the respective objective of a user.

### Claim 15 (depends on 9)

The computer-implemented system of claim 9, wherein the operations comprise performing an action based on the output data.

### Claim 16 (depends on 15)

The computer-implemented system of claim 15, wherein the action comprises reassigning one or more users to a different objective based on the output data.

### Claim 17

A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:
identifying, for a group of users in a shared virtual space, a respective objective for each of two or more users in the group of users;
for each of two or more users:
determining, based on inputs from multiple sensors having different input modalities, a respective intent of the user, wherein at least a portion of the multiple sensors are sensors of a device of the user that enables the user to participate in the shared virtual space; and
determining, based on the respective intent of the user, whether the user is performing a respective objective for the user;
generating, for the group of users, output data based on a respective objective for each of the two or more users and a respective intent for each of the two or more users; and
providing, to a respective device of each of one or more users in the group of users, the output data for presentation at the respective device of each of the one or more users.

### Claim 18 (depends on 17)

The non-transitory, computer-readable medium of claim 17, wherein a respective objective for at least one user of two or more users, comprises at least one of (i) a task to be performed by at least one user or (ii) a subject object to which at least one user should be looking.

### Claim 19 (depends on 18)

The non-transitory, computer-readable medium of claim 18, wherein identifying a respective objective for each of the two or more users comprises determining, as the subject object, a target to which at least a threshold amount of users in the group of users is looking.

### Claim 20 (depends on 17)

The non-transitory, computer-readable medium of claim 17, wherein:
a respective device of each user comprises a wearable device; and
determining a respective intent of a user comprises:
receiving, from the wearable device of the user, gaze data specifying a gaze of the user, gesture data specifying a hand gesture of the user, and direction data specifying a direction in which the user is moving; and
determining, as the respective intent of the user, an intent of the user with respective to a target object based on the gaze data, the gesture data, and the direction data.
