# US 12,002,457 B1 · Action eligibility for natural language processing systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,457 B1
- Document ID: US12002457B1
- Status: Granted
- Kind: B1
- Application number: 16/834,853
- Filed: 2020-03-30
- Granted: 2024-06-04
- Earliest claimed: 2020-03-30
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Eli Heitzman; Shekar Reddy; Christopher Anthony James; Jyoti Chhabra; Sylvester-Jaron Dewey Ogletree
- Web page: https://patentgrep.com/patent/US12002457B1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/18: using natural language modelling
- G10L 17/00: Speaker identification or verification techniques
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system is provided for determining which user inputs correspond to eligible actions for a particular device type. The system determines a device type identifier corresponding to the device that receives the user input and an action to be performed in response to the user input. The system determines that the performed action corresponds to an eligible action type for the device type identifier, and performs further processing using data related to the corresponding user input. Different device type identifiers may be associated with different eligible action types. The system may evaluate multiple user inputs from different users and different devices.

## Claims

### Claim 1

A computer-implemented method comprising:
during a first time period:
receiving, from a first device, audio data representing a first utterance;
determining, using the audio data, a first profile identifier;
determining, using the audio data and spoken language understanding, first user input data representing the first utterance;
performing a first action in response to the first utterance;
determining a first device type identifier corresponding to the first device;
storing first data in a first data storage, the first data including the first user input data, the first profile identifier, the first device type identifier, and a first action identifier corresponding to the first action;
determining second user input data representing a second utterance associated with a second profile identifier and received from a second device;
performing a second action in response to the second utterance;
determining a second device type identifier corresponding to the second device; and
storing second data including the second user input data, the second profile identifier, the second device type identifier, and a second action identifier corresponding to the second action;
during a second time period after the first time period:
determining a first eligible action type associated with the first device type identifier, the first eligible action type corresponding to an action that is eligible for reporting;
determining, using the first data, a first eligibility indication based on the first action corresponding to the first eligible action type;
determining a second eligible action type associated with the second device type identifier;
determining, using the second data, a second eligibility indication based on the second action corresponding to an action type different than the second eligible action type; and
storing, based on the first eligibility indication and the second eligibility indication, third data in a second data storage for further processing, the third data including the first user input data, the first profile identifier, the first device type identifier, and the first action identifier.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
during the first time period:
receiving, from the first device, second audio data representing a third utterance;
determining, using the second audio data, a third profile identifier;
determining, using the second audio data and spoken language understanding, third user input data representing the third utterance;
performing a third action in response to the third utterance;
determining that the third action corresponds to the first action identifier; and
storing fourth data in the first data storage, the fourth data including the third user input data, the third profile identifier, the first device type identifier, and the first action identifier;
during the second time period:
determining, using the third data, a third eligibility indication based on:
determining the third action corresponds to the first eligible action type, and
determining that the third profile identifier is different than the first profile identifier; and
storing, based on the third eligibility indication, fifth data in the second data storage for further processing, the fifth data including the third user input data, the third profile identifier, the first device type identifier, and the first action identifier.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining that the first action is performed with respect to a third device,
wherein determining the first eligible action type comprises determining the first eligible action type associated with the first device type identifier and a first device type corresponding to the third device, and
wherein storing the third data comprises storing the third data further including an indication of the first device type;
determining third user input data from a fourth device;
performing a third action in response to the third user input data, the third action being performed with respect to the third device, the third action corresponding to the first action identifier;
determining a third device type identifier corresponding to the fourth device;
determining that the first eligible action type is associated with the third device type identifier and the first device type;
determining a third eligibility indication based on the third action corresponding to the first eligible action type;
storing fourth data in the second data storage, the fourth data including the third user input data, a third profile identifier, the third device type identifier, the indication of the first device type, and the first action identifier;
determining first outcome data based on the third data indicating the first device type identifier and the first device type; and
determining second outcome data different from the first outcome data based on the fourth data indicating the third device type identifier and the first device type.

### Claim 4

A computer-implemented method comprising:
receiving, from a first device, a first user input associated with a first profile identifier, the first user input being an utterance;
determining a device type identifier corresponding to a device type of the first device;
determining a first action identifier, wherein the first action identifier corresponds to a first action performed in response to the first user input;
storing first input data in a first data storage, wherein the first input data includes a representation of the first user input, the first profile identifier, the device type identifier, and the first action identifier;
determining an eligible action type corresponding to an action that is eligible for reporting based on the device type identifier;
determining, using the first input data, that the first action is of the eligible action type;
determining that performing the first action is an initial performance of the first action associated with the first profile identifier; and
storing, based on the first action being of the eligible action type and determining the initial performance of the first action associated with the first profile identifier, second input data in a second data storage for further processing, wherein the second input data includes at least the first profile identifier, the device type identifier, and the first action identifier.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, from a second device, a second user input associated with a second profile identifier;
determining a second device type identifier corresponding to the second device;
determining a second action performed in response to the second user input, the second action corresponding to the first action identifier;
determining that the eligible action type is associated with the second device type identifier;
determining that the second action corresponds to the eligible action type;
storing, based on the second action corresponding to the eligible action type, third input data in the second data storage, wherein the third input data includes at least the second profile identifier, the second device type identifier, and the first action identifier;
determining first outcome data corresponding to the eligible action type and the device type identifier; and
determining second outcome data different than the first outcome data, the second outcome data corresponding to the eligible action type and the second device type identifier.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining that the eligible action type indicates that eligibility is based on one user input corresponding to one profile identifier;
determining, using second data from the second data storage, a second profile identifier associated with the first action identifier; and
determining that the second profile identifier is different than the first profile identifier, and
wherein storing, based on the first action corresponding to the eligible action type, the second input data in the second data storage comprises, storing the second input data based on the first profile identifier being different than the second profile identifier.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining that the eligible action type corresponds to a first domain;
receiving, from a second device, a second user input associated with a second profile identifier;
determining a second device type identifier corresponding to the second device;
determining a second action performed in response to the second user input; and
storing third input data in the first data storage wherein the third input data includes a representation of the second user input, the second profile identifier, the second device type identifier, and a second action identifier corresponding to the second action, wherein the first data storage stores data corresponding to user inputs that are related to the first domain, and
wherein determining that the first action corresponds to the eligible action type further comprises determining that the first action corresponds to the eligible action type after storing the third input data in the first data storage.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining that the eligible action type corresponds to a first domain;
determining, using a first skill system, output data responsive to the first user input; and
determining the first action using the output data,
wherein determining that the first action corresponds to the eligible action type further comprises determining that the first action corresponds to the eligible action type after determining the output data and prior to processing a second user input using the first skill system.

### Claim 9 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, from the first device, a second user input associated with a second profile identifier;
determining a second action performed in response to the second user input, the second action corresponding to the first action identifier;
storing third input data in the first data storage, wherein the third input data includes a representation of the second user input, the second profile identifier, the device type identifier, and the first action identifier;
determining, using the third input data, that the second action corresponds to the eligible action type;
determining that the second profile identifier is different than the first profile identifier; and
storing, based on the second action corresponding to the eligible action type and the second profile identifier being different than the first profile identifier, fourth input data in the second data storage, wherein the fourth input data includes at least the second profile identifier, the device type identifier, and the first action identifier.

### Claim 10 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining that the first action is performed with respect to a second device;
determining a second device type identifier associated with the second device; and
determining the eligible action type associated with the device type identifier and the second device type identifier.

### Claim 11 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, from the first device, audio data;
processing the audio data using spoken language understanding to determine the first user input;
processing the audio data to determine user recognition data; and
determining the first profile identifier using the user recognition data.

### Claim 12 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining outcome data based on determining correspondence of the first input data with the eligible action type and the device type identifier;
determining time data representing when the first action is performed;
wherein storing the second input data in the second data storage further comprises storing the second input data including the outcome data and the time data.

### Claim 13

A system comprising:
at least one processor; and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
receive, from a first device, a first user input associated with a first profile identifier, the first user input being an utterance;
determine a device type identifier corresponding to a device type of the first device;
determine a first action identifier, wherein the first action identifier corresponds to a first action performed in response to the first user input;
store first input data in a first data storage that stores first data corresponding to multiple user inputs associated with multiple user profiles, wherein the first input data includes a representation of the first user input, the first profile identifier, the device type identifier, and the first action identifier corresponding to the first action;
determine an eligible action type corresponding to an action that is eligible for reporting based on the device type identifier;
determine, using the first input data, that the first action is of the eligible action type;
determine that performing the first action is an initial performance of the first action associated with the first profile identifier; and
store, based on the first action being of the eligible action type and determining the initial performance of the first action associated with the first profile identifier, second input data in a second data storage for further processing, wherein the second input data includes at least the first profile identifier, the device type identifier, and the first action identifier.

### Claim 14 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further cause the system to:
receive, from a second device, a second user input associated with a second profile identifier;
determine a second device type identifier corresponding to the second device;
determine a second action performed in response to the second user input, the second action corresponding to the first action identifier;
determine that the eligible action type is associated with the second device type identifier;
determine that the second action corresponds to the eligible action type;
store, based on the second action corresponding to the eligible action type, third input data in the second data storage, wherein the third input data includes at least the second profile identifier, the second device type identifier, and the first action identifier;
determine first outcome data based on determining correspondence of the first input data with the eligible action type and the device type identifier; and
determine second outcome data different than the first outcome data, the second outcome data based on determining correspondence of the third input data with the eligible action type and the second device type identifier.

### Claim 15 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further causes the system to:
determine that the eligible action type indicates that eligibility is based on one user input corresponding to one profile identifier;
determine, using second data from the second data storage, a second profile identifier associated with the first action identifier; and
determine that the second profile identifier is different than the first profile identifier, and
wherein the instructions that cause the system to store the second input data in the second data storage further causes the system to store the second input data based on the first profile identifier being different than the second profile identifier.

### Claim 16 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further causes the system to:
determine that the eligible action type corresponds to a first domain;
receive, from a second device, a second user input associated with a second profile identifier;
determine a second device type identifier corresponding to the second device;
determine a second action performed in response to the second user input; and
store third input data in the first data storage wherein the third input data includes a representation of the second user input, the second profile identifier, the second device type identifier, and a second action identifier corresponding to the second action, wherein the first data storage stores data corresponding to user inputs that are related to the first domain, and
wherein the instructions that cause the system to determine that the first action corresponds to the eligible action type further causes the system to determine that the first action corresponds to the eligible action type after storing the third input data in the first data storage.

### Claim 17 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further cause the system to:
determine that the eligible action type corresponds to a first domain;
determine, using a first skill system, output data responsive to the first user input; and
determine the first action using the output data,
wherein the instructions that cause the system to determine that the first action corresponds to the eligible action type further causes the system to determine that the first action corresponds to the eligible action type after determining the output data and prior to processing a second user input using the first skill system.

### Claim 18 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further cause the system to:
receive, from the first device, a second user input associated with a second profile identifier;
determine a second action performed in response to the second user input, the second action corresponding to the first action identifier;
store third input data in the first data storage, wherein the third input data includes a representation of the second user input, the second profile identifier, the device type identifier, and the first action identifier;
determine, using the third input data, that the second action corresponds to the eligible action type;
determine that the second profile identifier is different than the first profile identifier; and
store, based on the second action corresponding to the eligible action type and the second profile identifier being different than the first profile identifier, fourth input data in the second data storage, wherein the fourth input data includes at least the second profile identifier, the device type identifier, and the first action identifier.

### Claim 19 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further cause the system to:
determine that the first action is performed with respect to a second device;
determine a second device type identifier associated with the second device; and
determine the eligible action type associated with the device type identifier and the second device type identifier.

### Claim 20 (depends on 13)

The system of claim 13, wherein the instructions that, when executed by the at least one processor, further cause the system to:
receive, from the first device, audio data;
process the audio data using spoken language understanding to determine the first user input;
process the audio data to determine a user recognition data; and
determine the first profile identifier using the user recognition data.