Multimodal task execution and text editing for a wearable system

A system comprising:

• a first sensor of a wearable system configured to acquire first user input data in a first mode of input;
• a second sensor of the wearable system configured to acquire second user input data in a second mode of input, the second mode of input different from the first mode of input;
• and
• a hardware processor in communication with the first and second sensors, the hardware processor programmed to:
• receive multimodal inputs comprising the first user input data in the first mode of input and the second user input data in the second mode of input;
• identify a first set of candidate objects for interactions based on the first user input data;
• identify a second set of candidate objects for interactions based on the second user input data;
• select a first candidate object within the first set of candidate objects;
• determine a first confidence score for the first candidate object based on the first user input data, wherein the first confidence score is based on a proportional area for a first portion of the first candidate object that is in the user's field of view with respect to a second portion of the first candidate object that is outside of the user's field of view;
• determine a second confidence score for the first candidate object based on the second user input data;
• calculate an aggregated confidence score for the first candidate object based on at least the first confidence score and the second confidence score;
• determine a target virtual object from the first set of candidate objects and the second set of candidate objects based on a combination of the first user input data, the second user input data, and the aggregated confidence score;
• determine a user interface operation on the target virtual object based on at least one of the first user input data or the second user input data;
• and
• generate a multimodal input command which causes the user interface operation to be performed on the target virtual object.