# US 2021/0345012 A1 · Methods and devices for clarifying audible video content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0345012 A1
- Document ID: US20210345012A1
- Status: Published application
- Kind: A1
- Application number: 17/372,909
- Filed: 2021-07-12
- Published: 2021-11-04
- Earliest claimed: 2014-06-20
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ingrid McAulay Trollope; Ant Oztaskent; Yaroslav Volovich
- Web page: https://patentgrep.com/patent/US20210345012A1

## CPC

- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G06F 17/40: Data acquisition and logging (for input to computer G06F3/00)
- G06F 40/00: Handling natural language data (speech analysis or synthesis, speech recognition G10L)
- G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/235: Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N 21/2393: {involving handling client requests (scheduling or organising the servicing of application requests H04L67/60)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The various implementations described herein include methods, devices, and systems for clarifying media content. In one aspect, a method is performed at a client device that includes a microphone, memory, and one or more processors. The method includes: (1) receiving, via the microphone, audio content of a media content item playing on a second client device in proximity to the client device; (2) receiving, via the microphone, a verbal query from a user of the client device to clarify a portion of the media content item; (3) sending a request to a remote server system; (4) receiving from the remote server system information responsive to the verbal user query for the portion of the media content item; and (5) presenting the information to the user.

## Claims

### Claim 1

A method, comprising:
at a client device having a microphone, one or more processors, and memory:
while a media content item is playing on a second client device in proximity to the client device, receiving, via the microphone, a verbal query from a user of the client device;
in response to receiving the verbal query while the media content item is playing on the second client device, sending a request to a remote server system, the request including at least a portion of the media content item and the verbal query;
in response to sending the request, receiving, from the remote server system, information responsive to the verbal query for the portion of the media content item; and
presenting, using a display of the client device, a card interface of an application executing on the client device that presents the information responsive to the verbal query to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the verbal query is a query regarding the portion of the media content item.

### Claim 3 (depends on 1)

The method of claim 1, wherein the verbal query is a request to clarify the portion of the media content item.

### Claim 4 (depends on 1)

The method of claim 1, wherein the verbal query concerns the media content item that is playing on the second client device.

### Claim 5 (depends on 1)

The method of claim 1, wherein the at least the portion of the media content is audio content.

### Claim 6 (depends on 1)

The method of claim 1, wherein the information comprises song lyrics.

### Claim 7 (depends on 1)

The method of claim 1, wherein the information comprises a transcription of speech.

### Claim 8 (depends on 1)

The method of claim 1, wherein the portion of the media content item corresponds to a plurality of persons speaking; and wherein the verbal query comprises a request to clarify the speech of a particular person of the plurality of persons.

### Claim 9 (depends on 1)

The method of claim 1, wherein the information comprises subtitle data associated with the media content item.

### Claim 10 (depends on 1)

The method of claim 1, wherein the information comprises information obtained from an online document.

### Claim 11 (depends on 1)

The method of claim 1, wherein the portion of the media content item includes speech in a first language; and wherein the information comprises a translation of the speech to a second language.

### Claim 12 (depends on 1)

The method of claim 1, wherein the verbal query comprises a request to translate content within the portion of the media content item; and wherein the information comprises a translation of the content.

### Claim 13 (depends on 1)

The method of claim 1, wherein the media content item is live television content.

### Claim 14 (depends on 1)

The method of claim 1, wherein the media content item is previously broadcast television content.

### Claim 15 (depends on 1)

The method of claim 1, wherein the media content item is recorded content.

### Claim 16 (depends on 1)

The method of claim 1, wherein the media content item is streaming content.

### Claim 17

An electronic device, comprising:
a microphone;
memory;
one or more processors; and
one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for:
while a media content item is playing on a second client device in proximity to the electronic device, receiving, via the microphone, a verbal query from a user of the client device;
in response to receiving the verbal query while the media content item is playing on the second client device, sending a request to a remote server system, the request including at least a portion of the media content item and the verbal query;
in response to sending the request, receiving, from the remote server system, information responsive to the verbal query for the portion of the media content item; and
presenting, using a display of the electronic device, a card interface of an application executing on the electronic device that presents the information responsive to the verbal query to the user.

### Claim 18 (depends on 17)

The electronic device of claim 17, further comprising a display, wherein the information is presented to the user via the display.

### Claim 19 (depends on 17)

The electronic device of claim 17, further comprising a speaker, wherein the information is presented to the user via the speaker.

### Claim 20

A non-transitory computer-readable storage medium storing one or more programs to be executed by an electronic device with a microphone, memory, and one or more processors, the one or more programs comprising instructions for:
while a media content item is playing on a second client device in proximity to the electronic device, receiving, via the microphone, a verbal query from a user of the electronic device;
in response to receiving the verbal query while the media content item is playing on the second client device, sending a request to a remote server system, the request including at least a portion of the media content item and the verbal query;
in response to sending the request, receiving, from the remote server system, information responsive to the verbal query for the portion of the media content item; and
presenting, using a display of the electronic device, a card interface of an application executing on the electronic device that presents the information responsive to the verbal query to the user.

### Claim 21 (depends on 20)

The non-transitory computer-readable storage medium of claim 20, wherein the portion of the media content item corresponds to a plurality of persons speaking; and wherein the verbal query comprises a request to clarify the speech of a particular person of the plurality of persons.
