# US 2025/0165070 A1 · Multi-Finger Gesture based on Finger Manipulation Data and Extremity Tracking Data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165070 A1
- Document ID: US20250165070A1
- Status: Published application
- Kind: A1
- Application number: 19/027,964
- Filed: 2025-01-17
- Published: 2025-05-22
- Earliest claimed: 2020-09-22
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Aaron M. Burns; Adam G. Poulos; Arun Rakesh Yoganandan; Benjamin Hylak; Benjamin R. Blachnitzky; Nicolai Georg
- Web page: https://patentgrep.com/patent/US20250165070A1

## CPC

- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method is performed at an electronic device with one or more processors, a non-transitory memory, a display, an extremity tracking system, and a communication interface provided to communicate with a finger-wearable device. The method includes displaying a computer-generated object on the display. The method includes obtaining finger manipulation data from the finger-wearable device via the communication interface. The method includes determining a multi-finger gesture based on extremity tracking data from the extremity tracking system and the finger manipulation data. The method includes registering an engagement event with respect to the computer-generated object according to the multi-finger gesture.

## Claims

### Claim 1

A method comprising:
at an electronic device with one or more processors, a non-transitory memory, an extremity tracking system, a display, and a communication interface provided to communicate with an input device:
while displaying a computer-generated object on the display:
obtaining, from the input device, input data during a first time period;
generating, via the extremity tracking system, extremity tracking data by performing computer vision, wherein performing the computer vision includes determining a respective movement path of a first finger across a plurality of images, wherein the respective movement path of the first finger is from a first position in an extended reality (ER) environment to a second position of the ER environment during a second time period, wherein the first time period is at least partially overlapping with the second time period;
determining a gesture based on the input data and the extremity tracking data, wherein determining the gesture is based on the respective movement path of the first finger; and
registering an engagement event with respect to the computer-generated object according to the gesture.

### Claim 2 (depends on 1)

The method of claim 1, wherein the input data indicates contact of a second finger with a physical object.

### Claim 3 (depends on 2)

The method of claim 2, wherein the input data indicates a contact intensity of the contact of the second finger with the physical object.

### Claim 4 (depends on 2)

The method of claim 2, wherein the first finger is of a first hand and the second finger is of a second hand.

### Claim 5 (depends on 1)

The method of claim 1, wherein performing the computer vision includes determining a respective movement path of a second finger across a plurality of images, wherein the respective movement path of the second finger is from a third position in the extended reality (ER) environment to a fourth position of the ER environment during a second time period and determining the gesture is further based on the respective movement path of the second finger.

### Claim 6 (depends on 1)

The method of claim 1, wherein the input data indicates a respective movement path of a second finger from a third position of the ER environment to a fourth position of the ER environment during the first time period.

### Claim 7 (depends on 1)

The method of claim 1, wherein registering the engagement event includes selecting the computer-generated object.

### Claim 8 (depends on 1)

The method of claim 1, wherein registering the engagement event includes manipulating the computer-generated object.

### Claim 9 (depends on 1)

The method of claim 1, wherein manipulating the computer-generated object includes moving the computer-generated object in the ER environment.

### Claim 10 (depends on 1)

The method of claim 1, wherein, while obtaining the finger manipulation data, the finger-wearable device is not viewable on the display.

### Claim 11

An electronic device comprising:
one or more processors;
a non-transitory memory;
a display;
an extremity tracking system;
a communication interface provided to communicate with an input device; and
one or more programs, wherein the one or more programs are stored in the
non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
while displaying a computer-generated object on the display:
obtaining, from the input device, input data during a first time period;
generating, via the extremity tracking system, extremity tracking data by performing computer vision, wherein performing the computer vision includes determining a respective movement path of a first finger across a plurality of images, wherein the respective movement path of the first finger is from a first position in an extended reality (ER) environment to a second position of the ER environment during a second time period, wherein the first time period is at least partially overlapping with the second time period;
determining a gesture based on the input data and the extremity tracking data, wherein determining the gesture is based on the respective movement path of the first finger; and
registering an engagement event with respect to the computer-generated object according to the gesture.

### Claim 12 (depends on 11)

The device of claim 11, wherein the input data indicates contact of a second finger with a physical object.

### Claim 13 (depends on 12)

The device of claim 12, wherein the input data indicates a contact intensity of the contact of the second finger with the physical object.

### Claim 14 (depends on 12)

The device of claim 12, wherein the first finger is of a first hand and the second finger is of a second hand.

### Claim 15 (depends on 11)

The device of claim 11, wherein performing the computer vision includes determining a respective movement path of a second finger across a plurality of images, wherein the respective movement path of the second finger is from a third position in the extended reality (ER) environment to a fourth position of the ER environment during a second time period and determining the gesture is further based on the respective movement path of the second finger.

### Claim 16 (depends on 11)

The device of claim 11, wherein the input data indicates a respective movement path of a second finger from a third position of the ER environment to a fourth position of the ER environment during the first time period.

### Claim 17

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or more processors, a display, an extremity tracking system, and a communication interface provided to communicate with an input device, cause the electronic device to:
while displaying a computer-generated object on the display:
obtain, from the input device, input data during a first time period;
generate, via the extremity tracking system, extremity tracking data by performing computer vision, wherein performing the computer vision includes determining a respective movement path of a first finger across a plurality of images, wherein the respective movement path of the first finger is from a first position in an extended reality (ER) environment to a second position of the ER environment during a second time period, wherein the first time period is at least partially overlapping with the second time period;
determine a gesture based on the input data and the extremity tracking data, wherein determining the gesture is based on the respective movement path of the first finger; and
register an engagement event with respect to the computer-generated object according to the gesture.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the input data indicates contact of a second finger with a physical object.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the one or more programs, when executed, cause the electronic device to perform the computer vision by determining a respective movement path of a second finger across a plurality of images, wherein the respective movement path of the second finger is from a third position in the extended reality (ER) environment to a fourth position of the ER environment during a second time period and the one or more programs, when executed, cause the electronic device to determine the gesture further based on the respective movement path of the second finger.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the input data indicates a respective movement path of a second finger from a third position of the ER environment to a fourth position of the ER environment during the first time period.
