Contextual natural language processing
AssigneeAmazon Technologies, Inc.
InventorsAngeliki Metallinou, Rahul Goel, Vishal Ishwar
Life of this patent
Estimating the term…
Abstract
Multi-modal natural language processing systems are provided. Some systems are context-aware systems that use multi-modal data to improve the accuracy of natural language understanding as it is applied to spoken language input. Machine learning architectures are provided that jointly model spoken language input (“utterances”) and information displayed on a visual display (“on-screen information”). Such machine learning architectures can improve upon, and solve problems inherent in, existing spoken language understanding systems that operate in multi-modal contexts.
Claims21
(canceled)
A computer-implemented method comprising:
- as performed by a computing system comprising one or more computer processors configured to execute specific instructions,
- receiving text data representing a user input to a computing device;
- receiving contextual data associated with content displayed by the computing device when the user input to the computing device occurred;
- generating natural language understanding (“NLU”) input data for an NLU subsystem using the text data and the contextual data, wherein the NLU input data comprises:
- first input data comprising an embedding representation of at least a portion of the text data;
- and
- second input data representing content displayed by the computing device when the user input to the computing device occurred;
- and
- generating output data using the NLU subsystem, the first input data, and the second input data, wherein the NLU subsystem comprises a machine learning model configured to generate the output data from the first input data and the second input data.
A system comprising:
- computer-readable memory storing executable instructions;
- and
- one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
- receive text data representing a user input to a computing device;
- receive contextual data associated with content displayed by the computing device when the user input to the computing device occurred;
- generate natural language understanding (“NLU”) input data for an NLU subsystem using the text data and the contextual data, wherein the NLU input data comprises:
- first input data comprising an embedding representation of at least a portion of the text data;
- and
- second input data representing content displayed by the computing device when the user input to the computing device occurred;
- and
- generate output data using the NLU subsystem, the first input data, and the second input data, wherein the NLU subsystem comprises a machine learning model configured to generate the output data from the first input data and the second input data.
Dependencies are inferred from the claim text.