# US 11,908,472 B1 · Connected accessory for a voice-controlled device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,472 B1
- Document ID: US11908472B1
- Status: Granted
- Kind: B1
- Application number: 17/941,791
- Filed: 2022-09-09
- Granted: 2024-02-20
- Earliest claimed: 2016-11-11
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Derick Deller; Link Cornelius; Apoorv Naik; Zoe Adams; Aslan Appleman; Pete Klein
- Web page: https://patentgrep.com/patent/US11908472B1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/295: Named entity recognition
- G06T 13/205: {driven by audio data}
- G10L 13/047: Architecture of speech synthesisers
- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 15/142: {Hidden Markov Models [HMMs]}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/025: for remote control or remote monitoring of applications
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/306: User profiles
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

Coordinated operation of a voice-controlled device and an accessory device in an environment is described. A remote system processes audio data it receives from the voice-controlled device in the environment to identify a first intent associated with a first domain, a second intent associated with a second domain, and a named entity associated with the audio data. The remote system sends, to the voice-controlled device, first information for accessing main content associated with the named entity, and a first instruction corresponding to the first intent. The remote system also sends, to the accessory device, second information for accessing control information or supplemental content associated with the main content, and a second instruction corresponding to the second intent. The first and second instructions, when processed by the devices in the environment, cause coordinated operation of the voice-controlled device and the accessory device.

## Claims

### Claim 1

A method comprising:
receiving, by a first voice interface device, audio data representing a user utterance;
determining, based at least in part on the audio data, that the user utterance corresponds to a request to perform a first action;
determining first data indicating a first device in proximity of the first voice interface device;
determining second data indicating the first device in proximity to a second voice interface device associated with the first voice interface device;
determining, based at least in part on the second data, that the second voice interface device last sent a first command to the first device; and
selecting the first device to perform the first action responsive to the request based at least in part on the second voice interface device last sending the first command to the first device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining an output type of output requested by the user utterance;
determining that the first device is associated with the output type; and
wherein selecting the first device comprises selecting the first device based at least in part on the first device being associated with the output type.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the request is to output a first portion of content by the first voice interface device while outputting a second portion of the content by the first device; and
selecting the first device comprises selecting the first device based at least in part on the request being to output the second portion of the content while the first portion of the content is output by the first voice interface device.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving a voice command associated with the first action at the second voice interface device;
determining an interaction between the second voice interface device and a second device; and
causing, based at least in part on the interaction, the second device to perform a second action while the first action is performed by the first device.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving a voice command associated with the first action at the second voice interface device;
determining an interaction between the second voice interface device and a second device; and
causing, based at least in part on the interaction, the second device to perform the first action.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
sending a second command to the first device based at least in part on selecting the first device;
determining that a response to the second command is not received within a predefined period of time from when the second command is sent;
determining that a second device is in communication with the first voice interface device; and
sending a third command to the second device based at least in part on the second device being in communication with the first voice interface device, the third command configured to cause the second device to perform the first action.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining that a second device is associated with the first device;
sending a second command to perform the first action to the first device based at least in part on selecting the first device; and
sending a third command to perform the first action to the second device based at least in part on the second device being associated with the first device.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining a first time indicator for when the audio data is received;
determining a second time indicator for when the first voice interface device last sent a prior command to the first device; and
wherein selecting the first device comprises selecting the first device based at least in part on the first time indicator and the second time indicator.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
determining that the first device has transitioned from a first location associated with the second voice interface device to a second location associated with the first voice interface device; and
wherein selecting the first device comprises selecting the first device based at least in part on the first device transitioning from the first location to the second location.

### Claim 10 (depends on 1)

The method of claim 1, wherein selecting the first device comprises selecting the first device based at least in part on a device type of the first device.

### Claim 11

A system comprising:
one or more processors; and
non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, by a first voice interface device, audio data representing a user utterance;
determining, based at least in part on the audio data, that the user utterance corresponds to a request to perform a first action;
determining first data indicating a first device in proximity of the first voice interface device;
determining second data indicating the first device in proximity to a second voice interface device associated with the first voice interface device;
determining, based at least in part on the second data, that the second voice interface device last sent a first command to the first device; and
selecting the first device to perform the first action responsive to the request based at least in part on the second voice interface device last sending the first command to the first device.

### Claim 12 (depends on 11)

The system of claim 11, the operations further comprising:
determining an output type of output requested by the user utterance;
determining that the first device is associated with the output type; and
wherein selecting the first device comprises selecting the first device based at least in part on the first device being associated with the output type.

### Claim 13 (depends on 11)

The system of claim 11, wherein:
the request is to output a first portion of content by the first voice interface device while outputting a second portion of the content by the first device; and
selecting the first device comprises selecting the first device based at least in part on the request being to output the second portion of the content while the first portion of the content is output by the first voice interface device.

### Claim 14 (depends on 11)

The system of claim 11, the operations further comprising:
receiving a voice command associated with the first action at the second voice interface device;
determining and interaction between the second voice interface device and to a second device; and
causing, based at least in part on the interaction, the second device to perform a second action while the first action is performed by the first device.

### Claim 15 (depends on 11)

The system of claim 11, the operations further comprising:
receiving a voice command associated with the first action at a second voice interface device;
determining an interaction between the second voice interface device and to a second device; and
causing, based at least in part on the interaction, the second device to perform the first action.

### Claim 16 (depends on 11)

The system of claim 11, the operations further comprising:
sending a second command to the first device based at least in part on selecting the first device;
determining that a response to the second command is not received within a predefined period of time from when the second command is sent;
determining that a second device is in communication with the first voice interface device; and
sending a third command to the second device based at least in part on the second device being in communication with the first voice interface device, the third command configured to cause the second device to perform the first action.

### Claim 17 (depends on 11)

The system of claim 11, the operations further comprising:
determining that a second device is associated with the first device;
sending a second command to perform the first action to the first device based at least in part on selecting the first device; and
sending a third command to perform the first action to the second device based at least in part on the second device being associated with the first device.

### Claim 18 (depends on 11)

The system of claim 11, the operations further comprising:
determining a first time indicator for when the audio data is received;
determining a second time indicator for when the first voice interface device last sent a prior command to the first device; and
wherein selecting the first device comprises selecting the first device based at least in part on the first time indicator and the second time indicator.

### Claim 19 (depends on 11)

The system of claim 11, the operations further comprising:
determining that the first device has transitioned from a first location associated with the second voice interface device to a second location associated with the first voice interface device; and
wherein selecting the first device comprises selecting the first device based at least in part on the first device transitioning from the first location to the second location.

### Claim 20 (depends on 11)

The system of claim 11, wherein selecting the first device comprises selecting the first device based at least in part on a functionality associated with the request.