US 2025/0149028 A1 · NATURAL LANGUAGE INTERACTIONS WITH INTERACTIVE VISUAL CONTENT

Natural language interactions with interactive visual content

AssigneeAmazon Technologies, Inc.

InventorsAmitabh Saikia, Devesh Mohan Pandey, Tagyoung Chung, Shanchan Wu, Chien-Wei Lin, Govindarajan Sundaram Thattai, Aishwarya Naresh Reganti, Arindam Mandal, Prakash Krishnan, Raefer Christopher Gabriel, Meyyappan Sundaram

Life of this patent

Estimating the term…

Abstract

Techniques for facilitating natural language interactions with visual interactive content are described. During a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. The learned information is used to store configuration data. During runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. The system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. Such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.

Claims21

Dependent claims

1-20. (canceled)

A computer-implemented method, comprising:

accessing a first website to determine first website data corresponding to the first website, the first website data indicating at least one web element operable to perform a first action using the first website;
processing the first website data to configure a first system component to, in response to a user input, interact with the first website data to provide a response to the user input;
after configuration of the first system component, receiving first input data corresponding to a first natural language input;
processing the first input data to determine the first natural language input corresponds to execution of the first action using the first website;
performing processing using the first system component to determine execution of the first action requires first information;
determining first output data representing a request for input of the first information;
causing presentation of the request using the first output data;
receiving second input data;
processing the second input data to determine a first value for the first information;
after determination of the first information, accessing the first website to provide the first value and cause execution of the first action;
after providing the first value, receiving, from the first website, second output data representing a result of the first action;
and
sending the second output data to a user device to cause presentation of information related to the result.

10 more elements

A system comprising:

at least one processor;
and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
access a first website to determine first website data corresponding to the first website, the first website data indicating at least one web element operable to perform a first action using the first website;
process the first website data to configure a first system component to, in response to a user input, interact with the first website data to provide a response to the user input;
after configuration of the first system component, receive first input data corresponding to a first natural language input;
process the first input data to determine the first natural language input corresponds to execution of the first action using the first website;
perform processing using the first system component to determine execution of the first action requires first information;
determine first output data representing a request for input of the first information;
cause presentation of the request using the first output data;
receive second input data;
process the second input data to determine a first value for the first information;
after determination of the first information, access the first website to provide the first value and cause execution of the first action;
after providing the first value, receive, from the first website, second output data representing a result of the first action;
and
send the second output data to a user device to cause presentation of information related to the result.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application

18/923,949Filed 23 Oct 2024

Priority

10 Dec 2021

Published

8 May 2025

Term ends

Not estimatedThe source records don’t support an estimate.

Classification9 codes

G10L 15/183
Using natural language modelling: using context dependencies, e.g. language models
G10L 15/22
Speech recognition: procedures used during a speech recognition process, e.g. man-machine dialogue
G06F 3/167
Sound input; Sound output: audio in a user interface, e.g. using voice commands for navigating, audio feedback
G10L 13/027
Methods for producing synthetic speech; Speech synthesisers: concept to speech synthesisers; Generation of natural phrases from machine-based concepts