# US 12,216,833 B1 · System for improving user input recognition on touch surfaces

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,833 B1
- Document ID: US12216833B1
- Status: Granted
- Kind: B1
- Application number: 18/355,006
- Filed: 2023-07-19
- Granted: 2025-02-04
- Earliest claimed: 2020-09-25
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Michele Stoppa; Waleed Abdulla; Henning Tjaden; Sree Harsha Kalli; Senem E. Emgin; John B. Morrell; Seung Wook Kim
- Web page: https://patentgrep.com/patent/US12216833B1

## CPC

- G06F 3/0219: {Special purpose keyboards}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 1/1662: {Details related to the integrated keyboard}
- G06F 1/169: {the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes (G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0213: {Arrangements providing an integrated pointing device in a keyboard, e.g. trackball, mini-joystick (for pointing devices integrated in a laptop computer G06F1/169; joysticks G05G9/047; constructional details of pointing devices G06F3/033)}
- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A physical keyboard can be used to collect user input in a typing mode or in a tracking mode. To use a tracking mode, first movement data is detected for a hand of a user in relation to a physical keyboard at a first location. A determination is made that the first movement data is associated with a tracking movement. In response to determining that the movement type is associated with the tracking movement, a tracking mode is initiated. User input is provided based on the movement data and in accordance with the tracking mode. Contact data and non-contact data is used to determine a user intent, and a user instruction is processed based on the user intent.

## Claims

### Claim 1

A method comprising:
obtaining sensor data for a hand of a user in relation to a physical input device, comprising:
obtaining contact data from one or more contact sensors of the physical input device, and
obtaining non-contact data for the hand of the user from one or more non-contact sensors, wherein the one or more contact sensors are different than the one or more non-contact sensors;
classifying the sensor data as satisfying a tracking mode based on the contact data and the non-contact data; and
processing the sensor data to obtain first user input in accordance with the tracking mode.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining a portion of the hand of the user making contact with the physical input device; and
classifying the sensor data based on the portion of the user making contact with the physical input device.

### Claim 3 (depends on 1)

The method of claim 1, wherein classifying the sensor data as satisfying a tracking mode comprises receiving an indication that a tracking mode has been initiated by a user.

### Claim 4 (depends on 1)

The method of claim 1, wherein classifying the sensor data as satisfying a tracking mode comprises detecting that the hand of the user performs a predetermined hand gesture.

### Claim 5 (depends on 1)

The method of claim 1, wherein classifying the sensor data as satisfying a tracking mode comprises:
determining a confidence value associated with the classification of the sensor data; and
determining that the confidence value satisfies a confidence threshold.

### Claim 6 (depends on 5)

The method of claim 5, wherein the confidence value is determined in part based on a gaze direction of the user.

### Claim 7 (depends on 6)

The method of claim 6, wherein the confidence value is determined in part based on detected movement of one or more fingers of the hand.

### Claim 8

A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain sensor data for a hand of a user in relation to a physical input device, comprising:
obtaining contact data from one or more contact sensors of the physical input device, and
obtaining non-contact data for the hand of the user from one or more non-contact sensors, wherein the one or more contact sensors are different than the one or more non-contact sensors;
classify the sensor data as satisfying a tracking mode based on the contact data and the non-contact data; and
process the sensor data to obtain first user input in accordance with the tracking mode.

### Claim 9 (depends on 8)

The non-transitory computer readable medium of claim 8, further comprising computer readable code to:
determine a portion of the hand of the user making contact with the physical input device; and
classify the sensor data based on the portion of the user making contact with the physical input device.

### Claim 10 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to receive an indication that a tracking mode has been initiated by a user.

### Claim 11 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to detect that the hand of the user performs a predetermined hand gesture.

### Claim 12 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to:
determine a confidence value associated with the classification of the sensor data; and
determine that the confidence value satisfies a confidence threshold.

### Claim 13 (depends on 12)

The non-transitory computer readable medium of claim 12, wherein the confidence value is determined in part based on a gaze direction of the user.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the confidence value is determined in part based on detected movement of one or more fingers of the hand.

### Claim 15

A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain sensor data for a hand of a user in relation to a physical input device, comprising:
obtaining contact data from one or more contact sensors of the physical input device, and
obtaining non-contact data for the hand of the user from one or more non-contact sensors, wherein the one or more contact sensors are different than the one or more non-contact sensors;
classify the sensor data as satisfying a tracking mode based on the contact data and the non-contact data; and
process the sensor data to obtain first user input in accordance with the tracking mode.

### Claim 16 (depends on 15)

The system of claim 15, further comprising computer readable code to:
determine a portion of the hand of the user making contact with the physical input device; and
classify the sensor data based on the portion of the user making contact with the physical input device.

### Claim 17 (depends on 15)

The system of claim 15, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to receive an indication that a tracking mode has been initiated by a user.

### Claim 18 (depends on 15)

The system of claim 15, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to detect that the hand of the user performs a predetermined hand gesture.

### Claim 19 (depends on 15)

The system of claim 15, wherein the computer readable code to classify the sensor data as satisfying a tracking mode comprises computer readable code to:
determine a confidence value associated with the classification of the sensor data; and
determine that the confidence value satisfies a confidence threshold.

### Claim 20 (depends on 19)

The system of claim 19, wherein the confidence value is determined in part based on a gaze direction of the user.