# US 10,768,693 B2 · Multimodal task execution and text editing for a wearable system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,768,693 B2
- Document ID: US10768693B2
- Status: Granted
- Kind: B2
- Application number: 15/955,204
- Filed: 2018-04-17
- Published: 2018-10-25
- Granted: 2020-09-08
- Earliest claimed: 2017-04-19
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: James M. Powderly; Savannah Niles; Jennifer M. R. Devine; Adam C. Carlson; Jeffrey Scott Sommers; Praveen Babu J D
- Web page: https://patentgrep.com/patent/US10768693B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G02B 6/0076: {Stacked arrangements of multiple light guides of the same or different cross-sectional area}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 2027/0127: {comprising devices increasing the depth of field}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Examples of wearable systems and methods can use multiple inputs (e.g., gesture, head pose, eye gaze, voice, and/or environmental factors (e.g., location)) to determine a command that should be executed and objects in the three-dimensional (3D) environment that should be operated on. The multiple inputs can also be used by the wearable system to permit a user to interact with text, such as, e.g., composing, selecting, or editing text.

## Claims

### Claim 1

A system comprising:
a first sensor of a wearable system configured to acquire first user input data in a first mode of input;
a second sensor of the wearable system configured to acquire second user input data in a second mode of input, the second mode of input different from the first mode of input; and
a hardware processor in communication with the first and second sensors, the hardware processor programmed to:
receive multimodal inputs comprising the first user input data in the first mode of input and the second user input data in the second mode of input;
identify a first set of candidate objects for interactions based on the first user input data;
identify a second set of candidate objects for interactions based on the second user input data;
select a first candidate object within the first set of candidate objects;
determine a first confidence score for the first candidate object based on the first user input data, wherein the first confidence score is based on a proportional area for a first portion of the first candidate object that is in the user's field of view with respect to a second portion of the first candidate object that is outside of the user's field of view;
determine a second confidence score for the first candidate object based on the second user input data;
calculate an aggregated confidence score for the first candidate object based on at least the first confidence score and the second confidence score;
determine a target virtual object from the first set of candidate objects and the second set of candidate objects based on a combination of the first user input data, the second user input data, and the aggregated confidence score;
determine a user interface operation on the target virtual object based on at least one of the first user input data or the second user input data; and
generate a multimodal input command which causes the user interface operation to be performed on the target virtual object.

### Claim 2 (depends on 1)

The system of claim 1, wherein the multimodal inputs comprise at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice.

### Claim 3 (depends on 1)

The system of claim 1, wherein the user interface operation comprises at least one of selecting, moving, or resizing the target virtual object.

### Claim 4 (depends on 1)

The system of claim 1, wherein the hardware processor is further configured to determine at least one of: a target location, orientation, or movement for the target virtual object in the user interface operation.

### Claim 5 (depends on 4)

The system of claim 4, wherein to determine the target location for the target virtual object, the hardware processor is programmed to identify a workable surface in a physical environment for putting the target virtual object.

### Claim 6 (depends on 5)

The system of claim 5, wherein the workable surface is identified by:
calculating a distance function for points of interest (POIs) on a physical object in the physical environment;
eliminating one or more of the POIs outside of a planar tolerance; and
delineating the workable surface based on remaining POIs.

### Claim 7 (depends on 5)

The system of claim 5, wherein the hardware processor is programmed to automatically orient the target virtual object to match an orientation of the target location.

### Claim 8 (depends on 1)

The system of claim 1, wherein the user interface operation is determined based on the first user input data in the first input mode, and at least one of a subject or a parameter is determined based on a combination of the first mode of input and the second mode of input.

### Claim 9 (depends on 1)

The system of claim 1, wherein the first input mode comprises an indirect input mode based on location information of a user of the wearable system.

### Claim 10 (depends on 9)

The system of claim 9, wherein the hardware processor is programmed to identify a virtual object as the target virtual object from the first set of candidate objects and the second set of candidate objects in response to a determination that the object is within a threshold range of the user.

### Claim 11 (depends on 1)

The system of claim 1, wherein the user interface operation is associated with a virtual application and the virtual application is programmed to be more responsive to one of the first sensor or the second sensor.

### Claim 12 (depends on 1)

The system of claim 1, wherein to determine the target virtual object from the first set of candidate objects and the second set of candidate objects, the hardware processor is programmed to perform a tree-based analysis on the first set of candidate objects and the second set of the candidate objects based on the first user input data and the second user input data.

### Claim 13 (depends on 1)

The system of claim 1, wherein to determine the target virtual object, the hardware processor is programmed to calculate a first or second confidence score for a virtual object by calculating at least one of:
an evenness of space around the virtual object in the field of view;
a proportional area for a first portion of the virtual object that is in the user's field of view with respect to a second portion of the virtual object that is outside of the user's field of view; or
a historical analysis of user's interactions with the virtual object.

### Claim 14 (depends on 1)

The system of claim 1, wherein the hardware processor is further programmed to: detect an initiation condition for an interaction event which triggers the hardware processor to determine the target virtual object and the user interface operation based on the multimodal inputs.

### Claim 15 (depends on 14)

The system of claim 14, wherein the initiation condition comprises a triggering phrase.

### Claim 16 (depends on 1)

The system of claim 1, wherein the first mode of input is a primary input mode and the second mode of input is a secondary input mode, and the hardware processor is programmed to: resolve ambiguities in at least one of the target virtual object and the user interface operation based on the second user input data.

### Claim 17 (depends on 1)

The system of claim 1, wherein the first user input data comprises a deictic or anaphoric reference to a virtual object and the hardware processor is programmed to identify the virtual object as the target virtual object based on the second user input data.

### Claim 18 (depends on 1)

The system of claim 1, wherein the hardware processor is further programmed to automatically enable, disable, or adjust a sensitivity of the first mode of input, the second mode of input, or both based at least in part on a user setting or an environment of the user.

### Claim 19 (depends on 1)

The system of claim 1, wherein the hardware processor is programmed to identify that target virtual object outside of a field of view of the user based at least in part on the multimodal inputs; and automatically move the target virtual object inside the field of view for user interaction.

### Claim 20 (depends on 1)

The system of claim 1, wherein the first confidence score is based on an evenness of space around the first candidate object in the field of view.

### Claim 21

A method comprising:
under control of a hardware processor of a wearable system in communication with a plurality of sensors configured to acquire user input data:
receiving the user input data from the plurality of sensors for an interaction event of a user with an environment;
analyzing the user input data to identify multimodal inputs for interacting with the environment wherein the multimodal inputs comprise a first input in a first input channel and a second input in a second input channel;
calculating, for a candidate object, a first confidence score correlating to the first input, wherein the first confidence score is based on a proportional area for a first portion of the candidate object that is in the user's field of view with respect to a second portion of the candidate object that is outside the user's field of view and a second confidence score correlating to the second input; and
calculating an aggregated score for the candidate object from at least the first confidence score and the second confidence;
determining, based at least in part on the aggregated score, a multimodal input command which comprises one or more of a subject, a parameter, or an operation for describing the interaction event with the environment, wherein at least one of the subject, the parameter, or the operation is identified based on a combination of the first input and the second input; and
causing the wearable system to execute the multimodal input command for the interaction event.

### Claim 22 (depends on 21)

The method of claim 21, wherein the operation comprises at least one of selecting, moving, or resizing the subject; wherein the subject comprises a target virtual object with which the user is about to interact; or wherein the parameter comprises at least a target location, orientation, or movement of the subject.

### Claim 23 (depends on 22)

The method of claim 22, wherein the target location is determined by:
calculating a distance function for points of interest (POIs) on a physical object;
eliminating one or more of the POIs outside of a given planar tolerance; and
delineating a workable surface on the physical object based on remaining POIs.

### Claim 24 (depends on 22)

The method of claim 22, wherein the method comprises automatically orienting the target virtual object to match that of the target location.

### Claim 25 (depends on 21)

The method of claim 21, wherein the multimodal inputs comprise at least two of the following input modes: head pose, eye gaze, user input device, hand gesture, or voice.

### Claim 26 (depends on 25)

The method of claim 25, wherein the first input channel is voice and the second input channel is a head pose or hand gesture.

### Claim 27 (depends on 21)

The method of claim 21, wherein at least one of the operation, subject, or parameter is further identified based on environment or location information of the user.

### Claim 28 (depends on 27)

The method of claim 27, wherein the subject is selected from a group of objects within a threshold range of the user.

### Claim 29 (depends on 21)

The method of claim 21, wherein the interaction event is within a virtual application and the virtual application is programmed to be more responsive to one of the first input channel or the second input channel.

### Claim 30 (depends on 21)

The method of claim 21, wherein the subject is identified by analyzing the first input and the second input using a lattice tree analysis to identify a virtual object for setting as the subject.

### Claim 31 (depends on 21)

The method of claim 21, wherein a candidate virtual object is identified as the subject based on confidence scores of virtual objects in a field of view of the user; and wherein the confidence scores are calculated based on at least one of:
an evenness of space around the candidate virtual object in the field of view;
a proportional area for a first portion the candidate virtual object that is in the user's field of view with respect to a second portion of the candidate virtual object that is outside of the user's field of view; or
a historical analysis of user's interactions with the candidate virtual object.

### Claim 32 (depends on 21)

The method of claim 21, further comprising: detecting an initiation condition for the interaction event based on data received from one or more sensors of the plurality of sensors, wherein the initiation condition triggers the receiving, analyzing, determining and causing steps.

### Claim 33 (depends on 21)

The method of claim 21, wherein the first input from the first input channel is a primary input and the second input from the second input channel is a secondary input, and the method comprises:
parsing the first input to identify the subject, the parameter, and the operation, and
resolving ambiguities in at least one of the subject, the parameter, or the operation based on the second input to generate the multimodal input command.

### Claim 34 (depends on 21)

The method of claim 21, wherein the method further comprises automatically enabling, disabling, or adjusting a sensitivity an input channel based at least in part on a user setting or the environment of the user.

### Claim 35 (depends on 21)

The method of claim 21, wherein the method further comprises identifying a virtual object that is outside of a field of view of the user as the subject based at least in part on the user interaction; and automatically move the virtual object inside the field of view for user interaction.

### Claim 36 (depends on 21)

The method of claim 21, wherein the first input comprises a deictic or anaphoric reference as the subject and the method further comprising selecting a target object as the subject based on the second input.

### Claim 37 (depends on 21)

The method of claim 21, wherein the first confidence score is based on an evenness of space around the first candidate object in the field of view.