US 11,443,740 B1Granted

Content selection by natural language processing systems

AssigneeAmazon Technologies, Inc.

InventorsRahul Ravu, Ye He, Felix Xiaomeng Wu, John Dunavent

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for performing tests by a speech processing system are described. A speech processing system may apply a hash function to a group of user identifiers, a content provider identifier, and time data to segment the group of user identifiers into non-holdout, holdout control, and holdout treatment groups. A non-holdout group represents users not subject to a test for a duration of time. A holdout control group represents users subject to a test for the duration of time, but which do not receive unresponsive content during the duration of time. A holdout treatment group represents users subject to a test and which receive unresponsive content over the duration of time. User activity of the holdout control and holdout treatment groups may be compared to measure an effectiveness of the unresponsive content. The measured effectiveness may be used to determine a ranking of the content provider at runtime.

Claims20

Dependent claims

A method comprising:

during a first time period:
determining a plurality of user identifiers to potentially receive output content corresponding to a test of unresponsive content for a speech processing system;
determining a content system identifier corresponding to first content to be tested;
determining a timestamp corresponding to a beginning of a first test corresponding to the content system identifier;
and
using a function, processing the content system identifier, the plurality of user identifiers, and the timestamp to generate a holdout treatment group for the first test, the holdout treatment group comprising a subset of the plurality of user identifiers;
during a second time period after the first time period:
receiving, from a first device, first audio data representing a first utterance;
determining second content responsive to the first utterance;
determining a first user identifier associated with the first audio data;
determining unsolicited content, corresponding to a topic different than the first utterance, is to be output;
determining the content system identifier is to be used to output the first content as the unsolicited content;
determining the first user identifier is represented in the holdout treatment group;
sending, to the first device, the second content for output;
based at least in part on determining the first user identifier is represented in the holdout treatment group, send, to the first device, the first content for output;
after sending the first content to the first device, receiving, from the first device, second audio data representing a second utterance;
and
storing system usage data associating the second utterance and the content system identifier;
during a third time period after the second time period:
based at least in part on the system usage data, generating updated ranking data indicating how the content system identifier is ranked when determining what content system to invoke at runtime.

17 more elements

A method comprising:

processing a content system identifier and a first user identifier to determine first data representing the first user identifier is to be used to output first content during a first period of time;
after determining the first user identifier is to be used to output the first content during the first period of time, receiving, from a first device, first input data representing a natural language input;
determining second content responsive to the natural language input;
determining the first user identifier is associated with the first input data;
determining the content system identifier is to be used to output the first content in coordination with the second content, wherein the first content is unsolicited by the natural language input;
determining the first device is to output the first content based at least in part on:
the first data,
determining the first user identifier is associated with the first input data, and
determining the content system identifier is to be used to output the first content in coordination with the second content;
sending, to the first device, the first content for output;
and
sending, to the first device, the second content for output.

9 more elements

A system comprising:

at least one processor;
and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
process a content system identifier and a first user identifier to determine first data representing the first user identifier is to be used to output first content during a first period of time;
after determining the first user identifier is to be used to output the first content during the first period of time, receive, from a first device, first input data representing a natural language input;
determine second content responsive to the natural language input;
determine the first user identifier is associated with the first input data;
determine the content system identifier is to be used to output the first content in coordination with the second content, wherein the first content is unsolicited by the natural language input;
determine the first device is to output the first content based at least in part on:
the first data,
determining the first user identifier is associated with the first input data, and
determining the content system identifier is to be used to output the first content in coordination with the second content;
send, to the first device, the first content for output;
and
send, to the first device, the second content for output.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/711,727Filed 12 Dec 2019
Granted: 13 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification14 codes

G10L 15/22
Speech recognition: procedures used during a speech recognition process, e.g. man-machine dialogue
G06F 3/167
Sound input; Sound output: audio in a user interface, e.g. using voice commands for navigating, audio feedback
G06Q 10/10
Administration; Management: office automation; Time management
G06Q 30/015
Customer relationship services: providing customer assistance, e.g. assisting a customer within a business location or via helpdesk

Family

No related US applications are recorded in the indexed filing history.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key