Multimodal state circulation
AssigneeGoogle Inc.
InventorsShir Judith Yehoshua, David Kliger Elson, David P. Whipp
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request to perform a task from a user device; obtaining a dialog corresponding to the task; providing multiple protocol buffers to the user device; receiving a voice input and one or more annotated protocol buffers from the user device, the one or more annotated protocol buffers identifying corresponding non-verbal responses to content in the protocol buffers; and using the received protocol buffers to update a state of the dialog and to interpret the voice input.
Claims21
A method comprising:
- receiving a request from a user device to perform a task;
- obtaining a multimodal dialog corresponding to the task, wherein the multimodal dialog includes a plurality of prompts for input from a user;
- providing the plurality of prompts for the multimodal dialog to the user device;
- receiving, in a bundle, a plurality of responses from the user device representing a local state of multimodal dialog of the user device, the bundle comprising:
- a voice response to a prompt, and
- one or more non-voice responses to corresponding prompts that precede, in the multimodal dialog, the prompt resulting in the voice response;
- and
- using the one or more non-voice responses in the bundle to update a state of the multimodal dialog and to interpret the voice response.
A system comprising:
- one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- receiving a request from a user device to perform a task;
- obtaining a multimodal dialog corresponding to the task, wherein the multimodal dialog includes a plurality of prompts for input from a user;
- providing the plurality of prompts for the multimodal dialog to the user device;
- receiving, in a bundle, a plurality of responses from the user device representing a local state of multimodal dialog of the user device, the bundle comprising:
- a voice response to a prompt, and
- one or more non-voice responses to corresponding prompts that precede, in the multimodal dialog, the prompt resulting in the voice response;
- and
- using the one or more non-voice responses in the bundle to update a state of the multimodal dialog and to interpret the voice response.
One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
- receiving a request from a user device to perform a task;
- receiving a request from a user device to perform a task;
- obtaining a multimodal dialog corresponding to the task, wherein the multimodal dialog includes a plurality of prompts for input from a user;
- providing the plurality of prompts for the multimodal dialog to the user device;
- receiving, in a bundle, a plurality of responses from the user device representing a local state of multimodal dialog of the user device, the bundle comprising:
- a voice response to a prompt, and
- one or more non-voice responses to corresponding prompts that precede, in the multimodal dialog, the prompt resulting in the voice response;
- and
- using the one or more non-voice responses in the bundle to update a state of the multimodal dialog and to interpret the voice response.
Dependencies are inferred from the claim text.