Natural language processing with contextual data associated with content displayed by a computing device
AssigneeAmazon Technologies, Inc.
InventorsAngeliki Metallinou, Rahul Goel, Vishal Ishwar
Life of this patent
Estimating the term…
Abstract
Multi-modal natural language processing systems are provided. Some systems are context-aware systems that use multi-modal data to improve the accuracy of natural language understanding as it is applied to spoken language input. Machine learning architectures are provided that jointly model spoken language input (“utterances”) and information displayed on a visual display (“on-screen information”). Such machine learning architectures can improve upon, and solve problems inherent in, existing spoken language understanding systems that operate in multi-modal contexts.
Claims18
A computer-implemented method comprising:
- as performed by a computing system comprising one or more computer processors configured to execute specific instructions,
- receiving first data representing a user input to a computing device;
- determining one or more applications associated with content displayed by the computing device when the user input to the computing device occurred;
- receiving, from an application of the one or more applications, contextual data associated with the content;
- generating natural language understanding (“NLU”) data for an NLU subsystem using the first data and the contextual data, wherein the NLU data comprises:
- second data comprising an embedding representation of at least a portion of the first data;
- and
- third data comprising a first vector element representing first content associated with a first application of the one or more applications and a second vector element representing second content associated with a second application of the one or more applications;
- and
- generating output data using the NLU subsystem, the second data, and the third data, wherein the NLU subsystem comprises a machine learning model configured to generate the output data using the second data and the third data.
A system comprising:
- computer-readable memory storing executable instructions;
- and
- one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
- receive first data representing a user input to a computing device;
- determine one or more applications associated with content displayed by the computing device when the user input to the computing device occurred;
- receive, from an application of the one or more applications, contextual data associated with the content;
- generate natural language understanding (“NLU”) data for an NLU subsystem using the first data and the contextual data, wherein the NLU data comprises:
- second data comprising an embedding representation of at least a portion of the first data;
- and
- third data comprising a first vector element representing first content associated with a first application of the one or more applications and a second vector element representing second content associated with a second application of the one or more applications;
- and
- generate output data using the NLU subsystem, the second data, and the third data, wherein the NLU subsystem comprises a machine learning model configured to generate the output data using the second data and the third data.
Dependencies are inferred from the claim text.