# US 12,769,442 B2 · Hardware interaction proxy for accessing touchscreens

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,769,442 B2
- Document ID: US12769442B2
- Status: Granted
- Kind: B2
- Application number: 18/816,229
- Filed: 2024-08-27
- Published: 2025-03-06
- Granted: 2026-10-20
- Earliest claimed: 2023-08-29
- Assignees: The Regents of the University of Michigan
- Applicants: The Regents of The University of Michigan
- Inventors: Anhong Guo; Alanson Sample; Ruijie Geng; Thomas Krolikowski; Yasha Iravantchi; Chen Liang
- Web page: https://patentgrep.com/patent/US12769442B2

## CPC

- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/04162: {for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware}
- G06F 3/044: by capacitive means
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Touchscreen devices, designed with an assumed range of user abilities and interaction patterns, often present challenges for individuals with diverse abilities to operate independently. Prior efforts to improve accessibility through tools or algorithms necessitated alterations to touchscreen hardware or software, making them inapplicable for the large number of existing legacy devices. This disclosure introduces a hardware interaction proxy that performs physical interactions on behalf of users while allowing them to continue utilizing accessible interfaces, such as screen readers and assistive touch on smartphones, for interface exploration and command input. The proxy maintains an interface model for accurate target localization and utilizes exchangeable actuators for physical actuation across a variety of device types, effectively reducing user workload and minimizing the risk of mistouch. Evaluations reveal that this approach lowers the mistouch rate and empowers visually and motor impaired users to interact with otherwise inaccessible physical touchscreens more effectively.

## Claims

### Claim 1

A method for accessing a touchscreen using an interaction proxy device, comprising:
determining, by an interaction proxy device, content displayed on a touchscreen;
exposing, by the interaction proxy device, the content to a user of the interaction proxy device;
detecting, by the interaction proxy device, a sequence of inputs from the user, where each input in the sequence of inputs indicates a command for the touchscreen;
recording, by the interaction proxy device, the sequence of inputs in a script;
subsequently executing, by the interaction proxy device, the script in response to a trigger from the user; and
actuating an actuator of the interaction proxy device according to the script, thereby activating user interface elements of the touchscreen.

### Claim 2 (depends on 1)

The method of claim 1 further comprises determining, by the interaction proxy device, location of the touchscreen in relation to the interaction proxy device.

### Claim 3 (depends on 2)

The method of claim 2 wherein the location of the touchscreen is determined using at least one of a camera or a motion sensor.

### Claim 4 (depends on 2)

The method of claim 2 further comprises actuating an actuator in response to detecting the input and thereby activating the user interface element.

### Claim 5 (depends on 4)

The method of claim 4 wherein actuating the actuator includes supplying current to a solenoid in order to actuate the solenoid.

### Claim 6 (depends on 4)

The method of claim 4 wherein actuating the actuator includes supplying voltage to a capacitance circuit in order to activate the user interface element.