# US 2021/0280185 A1 · Interactive voice controlled entertainment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280185 A1
- Document ID: US20210280185A1
- Status: Published application
- Kind: A1
- Application number: 17/178,868
- Filed: 2021-02-18
- Published: 2021-09-09
- Earliest claimed: 2017-06-28
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Alfred Yong-Hock Tan; Matthew Luker; David Allen Markley
- Web page: https://patentgrep.com/patent/US20210280185A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 21/44222: {Analytics of user selections, e.g. selection of programmes or purchase activity (monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/279: Recognition of textual entities
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/57: for processing of video signals
- H04N 21/2187: Live feed
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4302: {Content synchronisation processes, e.g. decoder synchronisation}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4781: {Games}
- H04N 21/47815: {Electronic shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)}
- H04N 21/8126: {involving additional data, e.g. news, sports, stocks, weather forecasts}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems for receiving shouted-out user responses to broadcast entertainment content, and for determining the responsiveness of those responses in relation to the broadcast content. In particular, entertainment broadcasts can be accompanied by mark-up data that represents various events within a given broadcast, which can be compared to the shouted-out responses to determine their accuracy. For example, if a game show was broadcast and an individual started shouting out answers during the broadcast, embodiments disclosed herein could utilize a voice-controlled electronic device that captures the shouted-out answers and passes them on to a language processing system that determines whether they are correct by comparing the answers to the mark-up data. The voice-controlled electronic device can also “listen” to background sounds to capture the broadcast of the entertainment content, and send that content to the language processing system, which can use that captured data to synchronize the actual broadcast with the analysis of the shouted-out answers to provide individuals with an immersive entertainment experience.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A computer-implemented method, comprising:
operating a device in a first mode corresponding to sending audio data for language processing following detection of a wakeword;
receiving first audio data representing a first utterance;
processing the first audio data to determine a representation of the wakeword;
in response to determining the representation of the wakeword, causing language processing to be performed using at least a portion of the first audio data;
receiving an indication to operate the device in a second mode corresponding sending audio data for language processing without detection of a wakeword;
in response to receiving the indication, operating the device in the second mode;
receiving second audio data representing a second utterance; and
in response to operating the device in the second mode, causing language processing to be performed using at least a portion of the second audio data regardless of whether the second audio data includes a representation of the wakeword.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, further comprising:
determining the first utterance corresponds to a session involving a skill;
determining the session is to involve further utterances; and
in response to the session involving further utterances, generating the indication.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, further comprising:
determining the second utterance is directed to the device.

### Claim 24 (depends on 21)

The computer-implemented method of claim 21, further comprising:
determining a skill operating with respect to the device; and
operating the device in the second mode for utterances corresponding to the skill.

### Claim 25 (depends on 24)

The computer-implemented method of claim 24, further comprising:
receiving third audio data representing a third utterance not related to the skill; and
prior to causing language processing to be performed using at least a portion of the third audio data, determining the third audio data includes a representation of the wakeword.

### Claim 26 (depends on 24)

The computer-implemented method of claim 24, further comprising:
prior to causing the language processing to be performed using at least the portion of the second audio data, determining the second audio data corresponds to the skill.

### Claim 27 (depends on 24)

The computer-implemented method of claim 24, further comprising:
receiving third audio data representing a third utterance not related to the skill; and
discarding the third audio data.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, further comprising, by the device:
determining that the third utterance is not related to the skill.

### Claim 29 (depends on 24)

The computer-implemented method of claim 24, further comprising:
determining the skill has ceased operation with respect to the device; and
discontinuing operation of the device in the second mode.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, wherein the first utterance was spoken by a first user and the method further comprises:
receiving third audio data representing a third utterance spoken by a second user; and
prior to causing language processing to be performed using at least a portion of the third audio data, determining the third audio data includes a representation of the wakeword.

### Claim 31

A system comprising:
at least one processor; and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
operate a device in a first mode corresponding to sending audio data for language processing following detection of a wakeword;
receive first audio data representing a first utterance;
process the first audio data to determine a representation of the wakeword;
in response to determining the representation of the wakeword, cause language processing to be performed using at least a portion of the first audio data;
receive an indication to operate the device in a second mode corresponding sending audio data for language processing without detection of a wakeword;
in response to receiving the indication, operating the device in the second mode;
receive second audio data representing a second utterance; and
in response to operating the device in the second mode, cause language processing to be performed using at least a portion of the second audio data regardless of whether the second audio data includes a representation of the wakeword.

### Claim 32 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine the first utterance corresponds to a session involving a skill;
determine the session is to involve further utterances; and
in response to the session involving further utterances, generate the indication.

### Claim 33 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine the second utterance is directed to the device.

### Claim 34 (depends on 31)

The system of claim 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine a skill operating with respect to the device; and
operate the device in the second mode for utterances corresponding to the skill.

### Claim 35 (depends on 34)

The system of claim 34, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive third audio data representing a third utterance not related to the skill; and
prior to causing language processing to be performed using at least a portion of the third audio data, determine the third audio data includes a representation of the wakeword.

### Claim 36 (depends on 34)

The system of claim 34, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
prior to causing the language processing to be performed using at least the portion of the second audio data, determine the second audio data corresponds to the skill.

### Claim 37 (depends on 34)

The system of claim 34, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive third audio data representing a third utterance not related to the skill; and
discard the third audio data.

### Claim 38 (depends on 37)

The system of claim 37, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
cause the device to determine that the third utterance is not related to the skill.

### Claim 39 (depends on 34)

The system of claim 34, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine the skill has ceased operation with respect to the device; and
discontinue operation of the device in the second mode.

### Claim 40 (depends on 31)

The system of claim 31, the first utterance was spoken by a first user and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive third audio data representing a third utterance spoken by a second user; and
prior to causing language processing to be performed using at least a portion of the third audio data, determine the third audio data includes a representation of the wakeword.