# US 2019/0317725 A1 · Speech-based interaction with a display window

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317725 A1
- Document ID: US20190317725A1
- Status: Published application
- Kind: A1
- Application number: 16/451,954
- Filed: 2019-06-25
- Published: 2019-10-17
- Earliest claimed: 2017-01-05
- Assignees: Banma Zhixing Network (Hongkong) Co., Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Shanwen Shi
- Web page: https://patentgrep.com/patent/US20190317725A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Speech-based interaction with a display window is disclosed, including: receiving speech input with respect to a display window that is presented at a device; converting the speech input into an operation instruction; determining an interface element of the display window that corresponds to the operation instructions, including by querying interface element information corresponding to the display window; and activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window.

## Claims

### Claim 1

A device, comprising:
one or more processors configured to:
receive, at the device, speech input with respect to a display window that is presented at the device;
convert the speech input into an operation instruction;
determine an interface element of the display window that corresponds to the operation instructions, including by querying interface element information corresponding to the display window; and
activate the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window; and
one or more memories coupled to the one or more processors and configured to provide the one or more processors with instructions.

### Claim 2 (depends on 1)

The device of claim 1, wherein the one or more processors are further configured to:
monitor a plurality of changes that are performed by a window manager service in response to a plurality of manual user interactions with the display window; and
generate the interface element information corresponding to the display window based at least in part on the plurality of changes.

### Claim 3 (depends on 1)

The device of claim 1, wherein the speech input is received via a sound-based sensor.

### Claim 4 (depends on 1)

The device of claim 1, wherein the operation instruction comprises identifying information associated with the interface element and operation descriptive information.

### Claim 5 (depends on 1)

The device of claim 1, wherein to convert the speech input into the operation instruction comprises to apply speech recognition to the speech input to convert the speech input into text.

### Claim 6 (depends on 1)

The device of claim 1, wherein to activate the interface element of the display window comprises to:
determine a selection position corresponding to the interface element based at least in part on layout information associated with the display window; and
send a first manual selection event with respect to the selection position to at least a window manager service, wherein the window manager service is configured to convert the first manual selection event into a second manual selection event with respect to the interface element and wherein the window manager service is configured to send the second manual selection event to the display window.

### Claim 7 (depends on 6)

The device of claim 6, wherein the layout information associated with the display window is included in the interface element information corresponding to the display window.

### Claim 8 (depends on 6)

The device of claim 6, wherein to determine the selection position corresponding to the interface element based at least in part on the layout information associated with the display window comprises to:
use the layout information associated with the display window to determine a display region corresponding to the interface element; and
select a position within the display region corresponding to the interface element as the selection position corresponding to the interface element.

### Claim 9 (depends on 1)

The device of claim 1, wherein to activate the interface element of the display window comprises to send a simulated manual selection event to the display window.

### Claim 10 (depends on 1)

The device of claim 1, wherein the interface element comprises one or more of the following: a button, a list, a menu, and an input field.

### Claim 11

A method, comprising:
receiving speech input with respect to a display window that is presented at a device;
converting the speech input into an operation instruction;
determining an interface element of the display window that corresponds to the operation instructions, including by querying interface element information corresponding to the display window; and
activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
monitoring a plurality of changes that are performed by a window manager service in response to a plurality of manual user interactions with the display window; and
generating the interface element information corresponding to the display window based at least in part on the plurality of changes.

### Claim 13 (depends on 11)

The method of claim 11, wherein the speech input is received via a sound-based sensor.

### Claim 14 (depends on 11)

The method of claim 11, wherein the operation instruction comprises identifying information associated with the interface element and operation descriptive information.

### Claim 15 (depends on 11)

The method of claim 11, wherein converting the speech input into the operation instruction comprises applying speech recognition to the speech input to convert the speech input into text.

### Claim 16 (depends on 11)

The method of claim 11, wherein activating the interface element of the display window comprises:
determining a selection position corresponding to the interface element based at least in part on layout information associated with the display window; and
sending a first manual selection event with respect to the selection position to at least a window manager service, wherein the window manager service is configured to convert the first manual selection event into a second manual selection event with respect to the interface element and wherein the window manager service is configured to send the second manual selection event to the display window.

### Claim 17 (depends on 16)

The method of claim 16, wherein the layout information associated with the display window is included in the interface element information corresponding to the display window.

### Claim 18 (depends on 16)

The method of claim 16, wherein determining the selection position corresponding to the interface element based at least in part on the layout information associated with the display window comprises:
using the layout information associated with the display window to determine a display region corresponding to the interface element; and
selecting a position within the display region corresponding to the interface element as the selection position corresponding to the interface element.

### Claim 19 (depends on 11)

The method of claim 11, wherein activating the interface element of the display window comprises sending a simulated manual selection event to the display window.

### Claim 20

A computer program product, the computer program product being embodiment in a non-transitory transitory computer readable storage medium and comprising computer instructions for:
receiving speech input with respect to a display window that is presented at a device;
converting the speech input into an operation instruction;
determining an interface element of the display window that corresponds to the operation instructions, including by querying interface element information corresponding to the display window; and
activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window.
