# US 2026/0065908 A1 · Talk back from actions in applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065908 A1
- Document ID: US20260065908A1
- Status: Published application
- Kind: A1
- Application number: 19/289,617
- Filed: 2025-08-04
- Published: 2026-03-05
- Earliest claimed: 2017-05-18
- Assignees: Peloton Interactive, Inc.
- Applicants: Peloton Interactive, Inc.
- Inventors: Mark Robinson; Matan Levi; Kiran Bindhu Hemaraj; Rajat Mukherjee
- Web page: https://patentgrep.com/patent/US20260065908A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

## Claims

### Claim 1

(canceled)

### Claim 2

A non-transitory computer storage medium storing computer-useable instructions that, when used by at least one processor, cause the at least one processor to:
determine that a received command corresponds to an action associated with an application,
wherein the action includes a workflow of the application and an object that corresponds to a graphical user interface (GUI) object presented via the application at a specific state of the workflow;
emulate the workflow in response to the determination that the received command corresponds to the action; and
decode, at the specific state of the emulated workflow, the object to provide at least one output that corresponds to the object.

### Claim 3 (depends on 2)

The computer storage medium of claim 2, wherein the object is associated with metadata and the metadata is included in the at least one output.

### Claim 4 (depends on 2)

The computer storage medium of claim 2, wherein the object is decoded based on metadata associated with the object.

### Claim 5 (depends on 4)

The computer storage medium of claim 4, wherein the instructions further cause the at least one processor to:
determine an entity type of the object based on the metadata,
wherein a context associated with the determined entity type is included in the at least one output.

### Claim 6 (depends on 2)

The computer storage medium of claim 2, wherein the at least one output that corresponds to the object is provided to a remote computing device configured to audibly provide the at least one output.

### Claim 7 (depends on 2)

The computer storage medium of claim 2, wherein the at least one output that corresponds to the object is provided as input to at least one field of another application.

### Claim 8 (depends on 2)

The computer storage medium of claim 2, wherein the at least one output includes a verbal communication.

### Claim 9 (depends on 2)

The computer storage medium of claim 2, wherein the at least one output includes a visual presentation based on a determination that an audio capability is disabled.

### Claim 10 (depends on 2)

The computer storage medium of claim 2, wherein the at least one output includes a sound or a color that corresponds to the object.

### Claim 11

A computer-implemented method, comprising:
determining, by a computing device, that a received command corresponds to an action associated with an application,
wherein the action includes a workflow of the application and an object that corresponds to a graphical user interface (GUI) object presented via the application at a specific state of the workflow;
emulating, by the computing device, the workflow in response to the determination that the received command corresponds to the action; and
decoding, by the computing device, the object at the specific state of the emulated workflow based on metadata associated with the object.

### Claim 12 (depends on 11)

The method of claim 11, wherein the object provides at least one audible output that corresponds to the metadata.

### Claim 13 (depends on 12)

The method of claim 12, wherein the metadata corresponds to one or more of a unit of measurement, a date, a starting point and an ending point, or a template for constructing the at least one audible output.

### Claim 14 (depends on 11)

The method of claim 11, wherein the determination that the received command corresponds to the action associated with the application is based at least in part on obtained contextual data.

### Claim 15 (depends on 14)

The method of claim 14, wherein the command is received as a voice command.

### Claim 16

A computer-implemented method, comprising:
receiving, by a digital assistant executing on a mobile device, a talk back add command to add a talk back feature that corresponds to an output of an application installed on the mobile device,
initiating, by the digital assistant, a talk back recording mode based on the received talk back add command, wherein the digital assistant in the talk back recording mode is configured to record a set of events that corresponds to a workflow of the application;
analyzing, by the digital assistant, a view of the application at a specific state to identify objects within the view;
receiving, by the digital assistant, a selection of a talk back object selected from the identified objects, wherein the talk back object is an output of the application at the specific state;
generating, by the digital assistant, a talk back output corresponding to the application based on the recorded set of events and the selected talk back object.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, further comprising:
decoding the talk back object to provide a specific audible sound, wherein the talk back object is generated based on the specific audible sound.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16, further comprising:
determining an entity type for the talk back object.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the entity type is determined by the digital assistant based on metadata associated with the talk back object.

### Claim 20 (depends on 18)

The computer-implemented method of claim 18, wherein the talk back object is associated with a context that corresponds to the entity type.

### Claim 21 (depends on 16)

The computer-implemented method of claim 16, further comprising:
invoking a particular talk back feature responsive to a detected utterance that corresponds to a particular command of one or more commands.
