Multi-Finger Gesture based on Finger Manipulation Data and Extremity Tracking Data

An electronic device comprising:

• one or more processors;
• a non-transitory memory;
• a display;
• an extremity tracking system;
• a communication interface provided to communicate with an input device;
• and
• one or more programs, wherein the one or more programs are stored in the
• non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
• while displaying a computer-generated object on the display:
• obtaining, from the input device, input data during a first time period;
• generating, via the extremity tracking system, extremity tracking data by performing computer vision, wherein performing the computer vision includes determining a respective movement path of a first finger across a plurality of images, wherein the respective movement path of the first finger is from a first position in an extended reality (ER) environment to a second position of the ER environment during a second time period, wherein the first time period is at least partially overlapping with the second time period;
• determining a gesture based on the input data and the extremity tracking data, wherein determining the gesture is based on the respective movement path of the first finger;
• and
• registering an engagement event with respect to the computer-generated object according to the gesture.

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or more processors, a display, an extremity tracking system, and a communication interface provided to communicate with an input device, cause the electronic device to:

• while displaying a computer-generated object on the display:
• obtain, from the input device, input data during a first time period;
• generate, via the extremity tracking system, extremity tracking data by performing computer vision, wherein performing the computer vision includes determining a respective movement path of a first finger across a plurality of images, wherein the respective movement path of the first finger is from a first position in an extended reality (ER) environment to a second position of the ER environment during a second time period, wherein the first time period is at least partially overlapping with the second time period;
• determine a gesture based on the input data and the extremity tracking data, wherein determining the gesture is based on the respective movement path of the first finger;
• and
• register an engagement event with respect to the computer-generated object according to the gesture.