# US 2025/0149028 A1 · Natural language interactions with interactive visual content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0149028 A1
- Document ID: US20250149028A1
- Status: Published application
- Kind: A1
- Application number: 18/923,949
- Filed: 2024-10-23
- Published: 2025-05-08
- Earliest claimed: 2021-12-10
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Amitabh Saikia; Devesh Mohan Pandey; Tagyoung Chung; Shanchan Wu; Chien-Wei Lin; Govindarajan Sundaram Thattai; Aishwarya Naresh Reganti; Arindam Mandal; Prakash Krishnan; Raefer Christopher Gabriel; Meyyappan Sundaram
- Web page: https://patentgrep.com/patent/US20250149028A1

## CPC

- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for facilitating natural language interactions with visual interactive content are described. During a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. The learned information is used to store configuration data. During runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. The system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. Such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A computer-implemented method, comprising:
accessing a first website to determine first website data corresponding to the first website, the first website data indicating at least one web element operable to perform a first action using the first website;
processing the first website data to configure a first system component to, in response to a user input, interact with the first website data to provide a response to the user input;
after configuration of the first system component, receiving first input data corresponding to a first natural language input;
processing the first input data to determine the first natural language input corresponds to execution of the first action using the first website;
performing processing using the first system component to determine execution of the first action requires first information;
determining first output data representing a request for input of the first information;
causing presentation of the request using the first output data;
receiving second input data;
processing the second input data to determine a first value for the first information;
after determination of the first information, accessing the first website to provide the first value and cause execution of the first action;
after providing the first value, receiving, from the first website, second output data representing a result of the first action; and
sending the second output data to a user device to cause presentation of information related to the result.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the second input data represents a second natural language input and wherein the method further comprises performing natural language processing using the second input data to determine the first information.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, further comprising, prior to receiving the first input data:
accessing a second website to determine second website data corresponding to the second website, the second website data indicating at least one second web element operable to perform a second action using the second website; and
processing the second website data to further configure the first system component.

### Claim 24 (depends on 21)

The computer-implemented method of claim 21, further comprising:
receiving first content data corresponding to the first website, the first content data corresponding to interactive visual content,
wherein configuration of the first system component uses the first content data.

### Claim 25 (depends on 21)

The computer-implemented method of claim 21, wherein:
causing presentation of the request comprises causes presentation of the request using the user device.

### Claim 26 (depends on 21)

The computer-implemented method of claim 21, wherein:
the first website data further comprises image data corresponding to the first website; and
configuration of the first system component uses the image data.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, further comprising:
processing the image data using bounding box detection to determine image feature data, wherein configuration of the first system component uses the image feature data.

### Claim 28 (depends on 26)

The computer-implemented method of claim 26, further comprising:
processing the image data using computer vision processing to determine image feature data,
wherein configuration of the first system component uses the image feature data.

### Claim 29 (depends on 21)

The computer-implemented method of claim 21, further comprising:
processing the first website data to determine web elements mapping data representing a relationship of the at least one web element to at least one second web element,
wherein configuration of the first system component uses the web elements mapping data.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, further comprising:
processing the first website data to determine semantic data corresponding to at least the first action,
wherein configuration of the first system component uses the semantic data.

### Claim 31

A system comprising:
at least one processor; and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
access a first website to determine first website data corresponding to the first website, the first website data indicating at least one web element operable to perform a first action using the first website;
process the first website data to configure a first system component to, in response to a user input, interact with the first website data to provide a response to the user input;
after configuration of the first system component, receive first input data corresponding to a first natural language input;
process the first input data to determine the first natural language input corresponds to execution of the first action using the first website;
perform processing using the first system component to determine execution of the first action requires first information;
determine first output data representing a request for input of the first information;
cause presentation of the request using the first output data;
receive second input data;
process the second input data to determine a first value for the first information;
after determination of the first information, access the first website to provide the first value and cause execution of the first action;
after providing the first value, receive, from the first website, second output data representing a result of the first action; and
send the second output data to a user device to cause presentation of information related to the result.

### Claim 32 (depends on 31)

The system of claim 31, wherein the second input data represents a second natural language input and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to perform natural language processing using the second input data to determine the first information.

### Claim 33 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to, prior to receipt of the first input data:
access a second website to determine second website data corresponding to the second website, the second website data indicating at least one second web element operable to perform a second action using the second website; and
process the second website data to further configure the first system component.

### Claim 34 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive first content data corresponding to the first website, the first content data corresponding to interactive visual content,
wherein configuration of the first system component uses the first content data.

### Claim 35 (depends on 31)

The system of claim 31, wherein the instructions that cause the system to cause presentation of the request comprise instructions that, when executed by the at least one processor, cause the system to cause presentation of the request using the user device.

### Claim 36 (depends on 31)

The system of claim 31, wherein:
the first website data further comprises image data corresponding to the first website; and
configuration of the first system component uses the image data.

### Claim 37 (depends on 36)

The system of claim 36, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the image data using bounding box detection to determine image feature data, wherein configuration of the first system component uses the image feature data.

### Claim 38 (depends on 36)

The system of claim 36, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the image data using computer vision processing to determine image feature data,
wherein configuration of the first system component uses the image feature data.

### Claim 39 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the first website data to determine web elements mapping data representing a relationship of the at least one web element to at least one second web element,
wherein configuration of the first system component uses the web elements mapping data.

### Claim 40 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the first website data to determine semantic data corresponding to at least the first action,
wherein configuration of the first system component uses the semantic data.
