# US 2026/0086712 A1 · Keyboard decoding using touch sensing images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0086712 A1
- Document ID: US20260086712A1
- Status: Published application
- Kind: A1
- Application number: 19/339,103
- Filed: 2025-09-24
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Piyawat Lertvittayakumjorn; Shanqing Cai; Peng Dou; Sze Chit Ho; Shumin Zhai
- Web page: https://patentgrep.com/patent/US20260086712A1

## CPC

- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 11/26: Drawing of charts or graphs
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computing device may detect user input on a presence-sensitive screen. In response to detecting the input, the method obtains indications representing the input and generates a touch sensing image from these indications. Information extracted from the touch sensing image is then input into an artificial intelligence (AI) model. The computing device applies the AI model to the information extracted from the touch sensing image to generate a distribution of candidate keys and their corresponding scores based on the touch sensing image. From this distribution, the method selects an alphanumeric key, which is subsequently outputted to the device's user interface in response to the selection.

## Claims

### Claim 1

A method comprising:
detecting, by one or more processors of a computing device, user input at a presence-sensitive screen;
responsive to a detection of the user input at the presence-sensitive screen, obtaining, by the one or more processors, indications representative of the user input;
generating, by the one or more processors, a touch sensing image from the indications representative of the user input detected at the presence-sensitive screen;
inputting, by the one or more processors, information extracted from the touch sensing image into an artificial intelligence model;
applying, by the one or more processors, the artificial intelligence model to the information extracted from the touch sensing image to generate a distribution of candidate keys and candidate key scores for the candidate keys based on the touch sensing image;
selecting, by the one or more processors, an alphanumeric key from the distribution of candidate keys and candidate key scores; and
responsive to a selection of the alphanumeric key, outputting, by the one or more processors, the alphanumeric key selected to a user interface of the computing device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
transforming, by the one or more processors, the touch sensing image into a heatmap overlap vector;
inputting, by the one or more processors, the heatmap overlap vector into the artificial intelligence model; and
applying, by the one or more processors using the artificial intelligence model, logistic regression to the heatmap overlap vector to generate the distribution.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining, by the one or more processors, a touch centroid corresponding to the user input entered at a region of the presence-sensitive screen;
inputting, by the one or more processors, the information extracted from the touch sensing image as a first input into the artificial intelligence model in a form of a heatmap overlap vector; and
inputting, by the one or more processors, the touch centroid as a second input into the artificial intelligence model in a form of a single point coordinate location within a region of the presence-sensitive screen.

### Claim 4 (depends on 3)

The method of claim 3, wherein determining the touch centroid corresponding to the user input entered at the region of the presence-sensitive screen comprises one of:
determining the touch centroid from the user input entered at the region of the presence-sensitive screen; or
deriving the touch centroid from the touch sensing image.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
extracting, by the one or more processors, touch centroid vector features from the user input or from the touch sensing image;
combining, by the one or more processors, the touch centroid vector features with heatmap overlap vector features derived from the touch sensing image into a single combined feature vector; and
inputting, by the one or more processors, the single combined feature vector into the artificial intelligence model.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
applying, by the one or more processors, a softmax function to the distribution generated by the artificial intelligence model to normalize the distribution having a sum of all candidate key scores equal to 1.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the one or more processors, multiple images as a series of discrete events corresponding to the indications representative of the user input entered at a region of the presence-sensitive screen
generating, by the one or more processors, the touch sensing image from the multiple images;
extracting, by the one or more processors, the information from the touch sensing image to a heatmap overlap vector; and
inputting, by the one or more processors, the heatmap overlap vector into the artificial intelligence model.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
training, by the one or more processors, the artificial intelligence model using a training dataset;
generalizing, by the one or more processors, the artificial intelligence model to unseen input data which forms no part of the training dataset; and
generating, by the one or more processors using the artificial intelligence model, the distribution from the information extracted from the touch sensing image which form no part of the training dataset.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
applying, by the one or more processors, a language model to the distribution of candidate keys and candidate key scores;
generating, by the one or more processors using the language model, a single selected key from the distribution of candidate keys and candidate key scores, wherein the single selected key has a highest combined score from the AI model and the language model; and
outputting, by the one or more processors, the single selected key as the alphanumeric key selected to the user interface of the computing device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the one or more processors, with the indications representative of the user input, properties of the user input including at least one or more of interaction duration, interaction touch pressure, interaction touch size, interaction touch movement, interaction gesture direction, interaction handedness, interaction orientation, time between user interactions, prior selected keys from prior distributions of candidate keys and candidate key scores, and prior text corrections;
applying, by the one or more processors, a language model to the distribution of candidate keys and candidate key scores;
inputting, by the one or more processors, the properties into the language model in association with the distribution of candidate keys and candidate key scores;
generating, by the one or more processors using the language model, a single selected key from the distribution of candidate keys and candidate key scores based at least in part on the properties; and
outputting, by the one or more processors, the single selected key as the alphanumeric key selected to the user interface of the computing device.

### Claim 11 (depends on 1)

The method of claim 1, wherein the touch sensing image represents a two-dimensional spatial map of user touch interactions detected within a region of the presence-sensitive screen.

### Claim 12 (depends on 1)

The method of claim 1:
wherein the user interface of the computing device is a virtual keyboard; and
wherein the method further comprises:
determining the user input entered is a key tap on the virtual keyboard based at least in part on the touch sensing image for the user input satisfying a threshold duration of time;
selecting the alphanumeric key from the distribution of candidate keys and candidate key scores; and
outputting, by the one or more processors, the alphanumeric key selected to the virtual keyboard.

### Claim 13

A computing device comprising:
a presence-sensitive screen configured to detect user input; and
one or more processors configured to:
responsive to a detection of the user input at the presence-sensitive screen, obtain indications representative of the user input;
generate a touch sensing image from the indications representative of the user input detected at the presence-sensitive screen;
input information extracted from the touch sensing image into an artificial intelligence model;
apply the artificial intelligence model to the information extracted from the touch sensing image to generate a distribution of candidate keys and candidate key scores for the candidate keys based on the touch sensing image;
select an alphanumeric key from the distribution of candidate keys and candidate key scores; and
responsive to a selection of the alphanumeric key, output the alphanumeric key selected to a user interface of the computing device.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the one or more processors are further configured to:
transform the touch sensing image into a heatmap overlap vector;
input the heatmap overlap vector into the artificial intelligence model; and
apply, using the artificial intelligence model, logistic regression to the heatmap overlap vector to generate the distribution.

### Claim 15 (depends on 13)

The computing device of claim 13, wherein the one or more processors are further configured to:
determine a touch centroid corresponding to the user input entered at a region of the presence-sensitive screen;
input the information extracted from the touch sensing image as a first input into the artificial intelligence model in a form of a heatmap overlap vector; and
input the touch centroid as a second input into the artificial intelligence model in a form of a single point coordinate location within the region of the presence-sensitive screen.

### Claim 16 (depends on 13)

The computing device of claim 13, wherein to determine the touch centroid corresponding to the user input entered at the presence-sensitive screen, the one or more processors are further configured to:
determine the touch centroid from the user input entered at a region of the presence-sensitive screen or derive the touch centroid from the touch sensing image.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein the one or more processors are further configured to:
extract touch centroid vector features from the user input or from the touch sensing image;
combine the touch centroid vector features with heatmap overlap vector features derived from the touch sensing image into a single combined feature vector; and
input the single combined feature vector into the artificial intelligence model.

### Claim 18 (depends on 13)

The computing device of claim 13, wherein the one or more processors are further configured to:
apply a softmax function to the distribution generated by the artificial intelligence model to normalize the distribution having a sum of all candidate key scores equal to 1.

### Claim 19 (depends on 13)

The computing device of claim 13, wherein the one or more processors are further configured to:
apply a language model to the distribution of candidate keys and candidate key scores; and
generate, using the language model, a single selected key from the distribution of candidate keys and candidate key scores, wherein the single selected key has a highest combined score from the AI model and the language model.

### Claim 20

Non-transitory computer-readable storage media comprising instructions that, when executed, configure one or more processors of a computing device to:
detect user input at a presence-sensitive screen;
responsive to a detection of the user input at the presence-sensitive screen, obtain indications representative of the user input;
generate a touch sensing image from the indications representative of the user input detected at the presence-sensitive screen;
input information extracted from the touch sensing image into an artificial intelligence model;
apply the artificial intelligence model to the information extracted from the touch sensing image to generate a distribution of candidate keys and candidate key scores for the candidate keys based on the touch sensing image;
select an alphanumeric key from the distribution of candidate keys and candidate key scores; and
responsive to a selection of the alphanumeric key, output the alphanumeric key selected to a user interface of the computing device.
