# US 11,442,582 B1 · Virtual keypads for hands-free operation of computing devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,442,582 B1
- Document ID: US11442582B1
- Status: Granted
- Kind: B1
- Application number: 17/193,610
- Filed: 2021-03-05
- Published: 2022-09-08
- Granted: 2022-09-13
- Earliest claimed: 2021-03-05
- Assignees: Zebra Technologies Corporation
- Applicants: Zebra Technologies Corporation
- Inventors: Joydeep Chakraborty; Ankit Kumar; Mukesh Prabhakar; Akashdeep Parida
- Web page: https://patentgrep.com/patent/US11442582B1

## CPC

- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)}
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes: detecting, at a computing device coupled to an image sensor, an input request; in response to detecting the input request, capturing a reference image via the image sensor, the reference image containing a virtual keypad having a set of virtual keys; detecting, in the reference image, the set of virtual keys; capturing, via the image sensor, a key-press image of the virtual keypad; detecting, in the key-press image, a subset of the virtual keys; comparing the set of virtual keys and the subset of virtual keys to identify a key-press obscuring one of the virtual keys; and returning input data corresponding to the obscured virtual key.

## Claims

### Claim 1

A method, comprising:
detecting, at a computing device coupled to an image sensor, an input request;
in response to detecting the input request, capturing a reference image via the image sensor, the reference image containing a virtual keypad having a set of virtual keys;
detecting, in the reference image, the set of virtual keys;
capturing, via the image sensor, a key-press image of the virtual keypad;
detecting, in the key-press image, a subset of the virtual keys;
comparing the set of virtual keys and the subset of virtual keys to identify a key-press obscuring one of the virtual keys; and
returning input data corresponding to the obscured virtual key,
wherein capturing the reference image includes:
capturing a sequence of initial images prior to the reference image,
detecting respective subsets of the virtual keys in each of the initial images, and
determining that the set of virtual keys from reference image matches the subset of virtual keys detected from at least one of the initial images.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting the input request includes generating the input request at the computing device, via execution of an application; and
wherein returning the input data includes passing the input data to the application.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
controlling an output device coupled to the computing device to generate an input notification including the input data.

### Claim 4 (depends on 1)

The method of claim 1, further comprising: responsive to determining that the set of virtual keys from reference image matches the subset of virtual keys detected from at least one of the initial images, enabling an input mode prior to capturing the key-press image.

### Claim 5 (depends on 1)

The method of claim 1, wherein capturing the key-press image includes:
capturing a sequence of input images prior to the key-press image;
in each input image, detecting at least a subset of the virtual keys; and
determining, based on comparison of the subset of virtual keys detected in the key-press image and the subsets of virtual keys in the input images, that an input confirmation criterion is satisfied.

### Claim 6 (depends on 5)

The method of claim 5, wherein the input confirmation criterion includes a threshold number of frames in which the same virtual key is obscured.

### Claim 7 (depends on 1)

The method of claim 1, further comprising storing relative positions of the virtual keys in the reference image.

### Claim 8 (depends on 1)

The method of claim 1, further comprising, prior to capturing the reference image:
retrieving a stored virtual keypad layout; and
controlling a projector coupled to the computing device to project the virtual keypad onto a surface within a field of view of the image sensor, according to the virtual keypad layout.

### Claim 9

A computing device, comprising:
a processor; and
an interface coupling the processor to an image sensor;
wherein the processor is configured to:
detect an input request;
in response to detecting the input request, capture a reference image via the image sensor, the reference image containing a virtual keypad having a set of virtual keys;
detect, in the reference image, the set of virtual keys;
capture, via the image sensor, a key-press image of the virtual keypad;
detect, in the key-press image, a subset of the virtual keys;
compare the set of virtual keys and the subset of virtual keys to identify a key-press obscuring one of the virtual keys; and
return input data corresponding to the obscured virtual key,
wherein the processor is configured to capture the reference image by:
capturing a sequence of initial images prior to the reference image,
detecting respective subsets of the virtual keys in each of the initial images, and
determining that the set of virtual keys from reference image matches the subset of virtual keys detected from at least one of the initial images.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein the processor is configured to detect the input request by generating the input request via execution of an application; and to return the input data by passing the input data to the application.

### Claim 11 (depends on 9)

The computing device of claim 9, wherein the processor is further configured to:
controlling an output device coupled to the processor to generate an input notification including the input data.

### Claim 12 (depends on 9)

The computing device of claim 9, wherein the processor is further configured, responsive to determining that the set of virtual keys from reference image matches the subset of virtual keys detected from at least one of the initial images, to enable an input mode prior to capturing the key-press image.

### Claim 13 (depends on 9)

The computing device of claim 9, wherein the processor is configured to capture the key-press image by:
capturing a sequence of input images prior to the key-press image;
in each input image, detecting at least a subset of the virtual keys; and
determining, based on comparison of the subset of virtual keys detected in the key-press image and the subsets of virtual keys in the input images, that an input confirmation criterion is satisfied.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the input confirmation criterion includes a threshold number of frames in which the same virtual key is obscured.

### Claim 15 (depends on 9)

The computing device of claim 9, wherein the processor is configured to store relative positions of the virtual keys in the reference image.

### Claim 16 (depends on 9)

The computing device of claim 9, wherein the processor is further configured, prior to capturing the reference image, to:
retrieve a stored virtual keypad layout; and
control a projector coupled to the computing device to project the virtual keypad onto a surface within a field of view of the image sensor, according to the virtual keypad layout.

### Claim 17

A system, comprising:
a head-mounted display including an image sensor;
a controller coupled to the head-mounted display, the controller configured to:
detect an input request;
in response to detecting the input request, capture a reference image via the image sensor, the reference image containing a virtual keypad having a set of virtual keys;
detect, in the reference image, the set of virtual keys;
capture, via the image sensor, a key-press image of the virtual keypad;
detect, in the key-press image, a subset of the virtual keys;
compare the set of virtual keys and the subset of virtual keys to identify a key-press obscuring one of the virtual keys; and
return input data corresponding to the obscured virtual key,
wherein the controller is configured to capture the reference image by:
capturing a sequence of initial images prior to the reference image,
detecting respective subsets of the virtual keys in each of the initial images, and
determining that the set of virtual keys from reference image matches the subset of virtual keys detected from at least one of the initial images.

### Claim 18 (depends on 17)

The system of claim 17, wherein the head-mounted display includes an eyeglass frame supporting the image sensor.

### Claim 19 (depends on 17)

The system of claim 17, wherein the controller is further configured to render the input data on the head-mounted display.

### Claim 20 (depends on 17)

The system of claim 17, further comprising a mobile computing device containing the controller.