# US 10,504,513 B1 · Natural language understanding with affiliated devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,513 B1
- Document ID: US10504513B1
- Status: Granted
- Kind: B1
- Application number: 15/716,353
- Filed: 2017-09-26
- Granted: 2019-12-10
- Earliest claimed: 2017-09-26
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Timothy Thomas Gray; Michal Grzegorz Kurpanik; Jenny Toi Wah Lam; Sarveshwar Nigam; Shirin Saleem; Jonhenry A. Righter; Jeremy Richard Hill; Kavya Ravikumar; Joe Virgil Fernandez; Kynan Dylan Antos; Kelly James Vanee
- Web page: https://patentgrep.com/patent/US10504513B1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 17/00: Speaker identification or verification techniques
- G06F 40/30: Semantic analysis
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A dock device connects participating devices such as a tablet device and an audio activated device, allowing them to operate as a single device. These participating devices may be associated with different accounts, each account being associated with particular “speechlets” or data processing functions. A natural language understanding (NLU) system uses NLU models to process text obtained from an automatic speech recognition (ASR) system to determine a set of possible intents. A second set of possible intents may then be generated that is limited to those possible intents that correspond to the speechlets associated with the docked device. The intents within the second set of possible intents are ranked, and the highest ranked intent may be deemed to be the intent of the user. Command data corresponding to the highest ranked intent may be generated and used to perform the action associated with that intent.

## Claims

### Claim 1

A system comprising:
at least one memory storing computer-executable instructions; and
at least one processor in communication with the at least one memory, the at least one processor executing the computer-executable instructions to:
determine a first device is associated with a user account;
determine a first speechlet associated with the user account, wherein the first speechlet comprises a first set of data processing functions available to the first device;
determine a second device is associated with the user account;
determine a second speechlet associated with the user account, wherein the second speechlet comprises a second set of data processing functions available to the second device;
generate a speechlet set comprising the first set of data processing functions and the second set of data processing functions;
determine a third device that is associated with the first device;
determine an output capability that is indicative of a type of output that the third device is able to present;
receive first data from the first device;
process the first data using a first natural language understanding (NLU) model to determine a set of possible intents that are representative of intended actions as expressed in the first data that are available in the speechlet set;
based at least in part on the output capability, determine a first intent of the set of possible intents as a first ranked intent associated with performing an action;
generate command data corresponding to the first ranked intent; and
send the command data to another device.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
at least one memory storing second computer-executable instructions; and
at least one processor in communication with the at least one memory, the at least one processor executing the second computer-executable instructions to:
determine a dock device that is associated with the user account; and
wherein a rank of an intent in the set of possible intents is based at least in part on an output capability of the dock device.

### Claim 3

A system comprising:
at least one memory storing computer-executable instructions; and
at least one processor in communication with the at least one memory, the at least one processor executing the computer-executable instructions to:
determine a first device is associated with a user account;
determine a second device is associated with the user account;
determine speechlet data indicative of one or more speechlets available to the first device and the second device to process one or more intents;
determine a third device that is associated with the first device;
determine an output capability that is indicative of a type of output that the third device is able to present;
receive first data from one or more of the first device or the second device;
process the first data using a first natural language understanding (NLU) model to determine a set of possible intents that are representative of intended actions as expressed in the first data that are available to the one or more speechlets indicated by the speechlet data;
rank the set of possible intents;
based at least in part on the output capability, select, from the set of possible intents, a first ranked intent associated with performing an intended action; and
generate command data corresponding to the first ranked intent.

### Claim 4 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
determine the one or more speechlets comprise a first speechlet associated with the first device, wherein the first speechlet comprises a first set of data processing functions; and
determine the one or more speechlets comprise a second speechlet associated with the second device, wherein the second speechlet comprises a second set of data processing functions;
wherein the speechlet data comprises a third set of data processing functions from the first speechlet and from the second speechlet.

### Claim 5 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
determine a second natural language understanding (NLU) model associated with the first device;
determine a third NLU model associated with the second device; and
wherein the first NLU model comprises the second NLU model and the third NLU model.

### Claim 6 (depends on 3)

The system of claim 3, the at least one processor executing the computer-executable instructions to determine the speechlet data by executing instructions to:
send at least one command to present a user interface to one or more of the first device or the second device;
receive from the one or more of the first device or the second device, selection data indicative of designation of the user account as obtained with the user interface; and
determine one or more data processing functions associated with the user account.

### Claim 7 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
determine identity data indicative of an identity of a speaker as represented by audio data;
determine the user account is associated with the identity data;
determine one or more data processing functions associated with the user account; and
designate the one or more data processing functions associated with the user account as the speechlet data.

### Claim 8 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
determine the one or more speechlets comprise a first speechlet that is associated with the user account;
determine the one or more speechlets comprise a second speechlet that is associated with the user account; and
wherein the speechlet data comprises the first speechlet and the second speechlet.

### Claim 9 (depends on 3)

The system of claim 3, the at least one processor executing the computer-executable instructions to:
determine a dock identifier that is associated with one or more of the first device or the second device; and
the at least one processor executing the computer-executable instructions to determine the speechlet data by executing instructions to:
determine one or more data processing functions that are associated with the dock identifier, wherein the speechlet data comprises information indicative of availability of the one or more data processing functions.

### Claim 10 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
determine the one or more speechlets comprise a first speechlet available to the first device and a second speechlet available to the second device;
determine first data indicative of first content available to the user account by using the first speechlet;
determine second data indicative of second content available to the user account by using the second speechlet;
select the first content based at least in part on the first data and the second data;
determine data processing functions that are accessible to the user account; and
designate the data processing functions that are accessible to the user account as the speechlet data.

### Claim 11 (depends on 3)

The system of claim 3, the at least one processor further executing the computer-executable instructions to:
receive image data from the one or more of the first device or the second device;
determine a count of people represented in the image data;
determine the count exceeds a threshold value; and
responsive to the count exceeding the threshold value, the at least one processor further executing the computer-executable instructions to determine the speechlet data by executing instructions to:
determine the one or more speechlets comprise a first speechlet that is available to the first device;
determine the one or more speechlets comprise a second speechlet that is available to the first device; and
wherein the speechlet data comprises the first speechlet and the second speechlet.

### Claim 12 (depends on 3)

The system of claim 3, wherein each intent of the set of possible intents is associated with a confidence value; and further wherein the rank of the set of possible intents is based on the confidence value for each of the intents in the set of possible intents.

### Claim 13

A method comprising:
determining a first device is associated with a user account;
determining a second device is associated with the user account;
determining a third device that is associated with the first device;
determining speechlet data that is indicative of one or more data processing functions available to the first device and the second device to process one or more intents;
determining an output capability that is indicative of a type of output that the third device is able to present;
receiving first data from one or more of the first device or the second device;
processing the first data using one or more natural language understanding (NLU) models to determine a set of possible intents that are representative of intended actions as expressed in the first data that are available to the one or more data processing functions indicated by the speechlet data;
based at least in part on the output capability and from the set of possible intents, determining a first ranked intent associated with performing an intended action; and
generating command data corresponding to the first ranked intent.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
receiving user input indicative of the user account; and
determining information indicative of one or more data processing functions that are associated with the user account.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
determining identity data indicative of an identity of a speaker as represented by the first data;
determining the user account is associated with the identity data; and
determining information indicative of one or more data processing functions that are associated with the user account.

### Claim 16 (depends on 13)

The method of claim 13, further comprising:
determining the one or more data processing functions comprise a first speechlet that is available to the first device; and
determining the one or more data processing functions comprise a second speechlet that is available to the second device.

### Claim 17 (depends on 13)

The method of claim 13, further comprising:
determining a dock identifier that is indicative of a dock device; and
determining one or more data processing functions that are associated with the dock identifier; and
wherein the speechlet data is indicative of the one or more data processing functions that are associated with the dock identifier.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
determining first content data indicative of first content available to the first device;
determining second content data indicative of second content available to the second device;
selecting the first content based at least in part on the first content data and the second content data; and
wherein the speechlet data is indicative of one or more data processing functions that are accessible to the user account.

### Claim 19 (depends on 13)

The method of claim 13, further comprising:
sending the command data to a computing device associated with the one or more data processing functions;
performing the one or more data processing functions; and
presenting output using the one or more of the first device or the second device.

### Claim 20 (depends on 13)

The method of claim 13, further comprising:
determining a second natural language understanding (NLU) model associated with the first device;
determining a third NLU model associated with the second device; and
wherein the NLU model comprises the second NLU model and the third NLU model.

### Claim 21 (depends on 13)

The method of claim 13, wherein each intent of the set of possible intents is associated with a confidence value; and further wherein a rank of the set of possible intents is based on the confidence value for each of the intents in the set of possible intents.