Back to results
US 11,080,016 B2Granted
Speech-based interaction with a display window
AssigneeBanma Zhixing Network (Hongkong) Co., Limited
InventorShanwen Shi
Life of this patent
Estimating the term…
Abstract
Speech-based interaction with a display window is disclosed, including: receiving speech input with respect to a display window that is presented at a device; converting the speech input into an operation instruction; determining an interface element of the display window that corresponds to the operation instructions, including by querying interface element information corresponding to the display window; and activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window.
Claims18
A device, comprising:
- one or more processors configured to:
- monitor and record a plurality of changes that are performed by a window manager service in response to a plurality of manual user interactions with a display window;
- generate interface element information corresponding to the display window based at least in part on the recorded plurality of changes, wherein the interface element information includes which one or more interface elements are present within the display window;
- receive, at the device, speech input with respect to the display window that is presented at the device;
- convert the speech input into an operation instruction;
- determine an interface element of the display window that corresponds to the operation instruction, including by querying the interface element information corresponding to the display window;
- and
- activate the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window, including to:
- determine layout information corresponding to the interface element of the display window from the interface element information corresponding to the display window, wherein the layout information corresponding to the interface element describes a display region of the interface element;
- determine a selection position corresponding to the interface element based at least in part on the layout information corresponding to the display window, wherein the selection position comprises a position within the display region of the interface element;
- and
- send a first manual selection event with respect to the selection position to at least the window manager service, wherein the window manager service is configured to convert the first manual selection event into a second manual selection event with respect to the interface element and wherein the window manager service is configured to send the second manual selection event to the display window;
- and
- one or more memories coupled to the one or more processors and configured to provide the one or more processors with instructions.
11 more elements
A method, comprising:
- monitoring and recording a plurality of changes that are performed by a window manager service in response to a plurality of manual user interactions with a display window;
- generating interface element information corresponding to the display window based at least in part on the recorded plurality of changes, wherein the interface element information includes which one or more interface elements are present within the display window;
- receiving speech input with respect to the display window that is presented at a device;
- converting the speech input into an operation instruction;
- determining an interface element of the display window that corresponds to the operation instruction, including by querying the interface element information corresponding to the display window;
- and
- activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window, including by:
- determining layout information corresponding to the interface element of the display window from the interface element information corresponding to the display window, wherein the layout information corresponding to the interface element describes a display region of the interface element;
- determining a selection position corresponding to the interface element based at least in part on the layout information corresponding to the display window, wherein the selection position comprises a position within the display region of the interface element;
- and
- sending a first manual selection event with respect to the selection position to at least the window manager service, wherein the window manager service is configured to convert the first manual selection event into a second manual selection event with respect to the interface element and wherein the window manager service is configured to send the second manual selection event to the display window.
8 more elements
A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- monitoring and recording a plurality of changes that are performed by a window manager service in response to a plurality of manual user interactions with a display window;
- generating interface element information corresponding to the display window based at least in part on the recorded plurality of changes, wherein the interface element information includes which one or more interface elements are present within the display window;
- receiving speech input with respect to the display window that is presented at a device;
- converting the speech input into an operation instruction;
- determining an interface element of the display window that corresponds to the operation instruction, including by querying the interface element information corresponding to the display window;
- and
- activating the interface element of the display window based at least in part on simulating a manual selection event with respect to the interface element of the display window, including by:
- determining layout information corresponding to the interface element of the display window from the interface element information corresponding to the display window, wherein the layout information corresponding to the interface element describes a display region of the interface element;
- determining a selection position corresponding to the interface element based at least in part on the layout information corresponding to the display window, wherein the selection position comprises a position within the display region of the interface element;
- and
- sending a first manual selection event with respect to the selection position to at least the window manager service, wherein the window manager service is configured to convert the first manual selection event into a second manual selection event with respect to the interface element and wherein the window manager service is configured to send the second manual selection event to the display window.
8 more elements
Dependencies are inferred from the claim text.