US 11,887,589 B1Granted

Voice-based interactions with a graphical user interface

AssigneeAmazon Technologies, Inc.

InventorsSenthil Kumar Dayalan, Manikandan Thangarathnam, Sai Vinayak, Suraj Gopalakrishnan

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for voice-based interactions are described. In an example, a device presents a user interface on a display. The device starts an operational mode of the device. The operational mode restricts voice-based interactions with the user interface to a set of commands. The set of commands is defined in a language model that is stored on the device. Further, the device receives, at a microphone of the device, audio data corresponding to a natural language utterance and generates, from the audio data, text data that corresponds to the natural language utterance. The device determines, based at least in part on the language model, that semantics of the text data correspond to a command from the set of commands and presents, on the display, an outcome of performing the command.

Claims20

Dependent claims

A system, comprising:

a set of servers;
and
a device communicatively coupled with the set of servers and configured with computer-readable instructions to:
present a menu on a display;
receive, at a microphone of the device, first audio data indicating a wakeword and a first natural language utterance for interaction with the menu;
upon detecting the wakeword, send, to the set of servers, second audio data indicating the first natural language utterance;
receive, from the set of servers, instruction data about starting a first operational mode of the device, the first operational mode restricting, for a time period, first voice-based interactions via the microphone and not using the wakeword to a subset of a plurality of menu commands associated with content in view on the display, the menu commands defined in a language model that is stored on the device, wherein the device is configured to operate in a second operational mode that permits, for the time period, second voice-based interactions associated with the plurality of menu commands and using the wakeword;
receive, at the microphone and during the time period, third audio data indicating a second natural language utterance;
generate, from the third audio data, text data that corresponds to the second natural language utterance;
determine, by the language model, that semantics of the text data correspond to a menu command from the subset of the plurality of menu commands;
perform, the menu command;
and
present, on the display, an outcome of performing the menu command.

10 more elements

A method implemented by a device, the method comprising:

presenting a user interface on a display;
starting a first operational mode of the device, the first operational mode restricting first voice-based interactions with the device to a subset of a set of commands associated with content in view on the display, the set of commands associated with the user interface and defined in a language model that is stored on the device, wherein the device is configured to operate in a second operational mode that permits second voice-based interactions associated with the set of commands;
receiving, at a microphone of the device, audio data corresponding to a natural language utterance;
generating, from the audio data, text data that corresponds to the natural language utterance;
determining, based at least in part on the language model, that semantics of the text data correspond to a command from the subset of the set of commands;
and
presenting, on the display, an outcome of performing the command.

4 more elements

A device comprising:

a microphone;
a processor;
and
a memory storing a language model and computer-readable instructions that, upon execution by the processor, configure the device to:
present a user interface on a display;
start a first operational mode of the device, the first operational mode restricting first voice-based interactions with the device to a subset of a set of commands associated with content in view on the display, the set of commands associated with the user interface and defined in the language model, wherein the device is configured to operate in a second operational mode that permits second voice-based interactions associated with the set of commands;
receive, at the microphone, audio data corresponding to a natural language utterance;
generate, from the audio data, text data that corresponds to the natural language utterance;
determine, based at least in part on the language model, that semantics of the text data correspond to a command from the subset of the set of commands;
and
present, on the display, an outcome of performing the command.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/903,698Filed 17 Jun 2020
Granted: 30 Jan 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

G06F 3/0485
For the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a…:…
G10L 15/22
Speech recognition: procedures used during a speech recognition process, e.g. man-machine dialogue
G06F 3/0482
Based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like…:…
G06F 3/167
Sound input; Sound output: audio in a user interface, e.g. using voice commands for navigating, audio feedback