# US 2025/0094491 A1 · Structured Video Documents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094491 A1
- Document ID: US20250094491A1
- Status: Published application
- Kind: A1
- Application number: 18/961,038
- Filed: 2024-11-26
- Published: 2025-03-20
- Earliest claimed: 2022-03-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Johan Schalkwyk; Francoise Beaufays
- Web page: https://patentgrep.com/patent/US20250094491A1

## CPC

- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G10L 15/04: Segmentation; Word boundary detection
- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 16/7834: {using audio features}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/30: Semantic analysis
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method includes receiving a content feed that includes audio data corresponding to speech utterances and processing the content feed to generate a semantically-rich, structured document. The structured document includes a transcription of the speech utterances and includes a plurality of words each aligned with a corresponding audio segment of the audio data that indicates a time when the word was recognized in the audio data. During playback of the content feed, the method also includes receiving a query from a user requesting information contained in the content feed and processing, by a large language model, the query and the structured document to generate a response to the query. The response conveys the requested information contained in the content feed. The method also includes providing, for output from a user device associated with the user, the response to the query.

## Claims

### Claim 1

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving image data comprising an image frame;
processing the image data to generate a structured document, the structured document comprising creator-provided text recognized in the image frame;
receiving a query from a user requesting information contained in the image data; and
processing the query and the structured document to generate a response to the query, the response conveying the requested information contained in the image data.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise providing, for output from a user device associated with user, the response to the query.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein processing the query and the structured document to generate the response to the query comprises using a large language model to process the query and the structured document to generate the response to the query as output from the large language model.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the large language model comprises a pre-trained large language model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the pre-trained large language model generates the response to the query by performing few-shot learning using the structured document as context for the query.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein the large language model comprises an encoding/decoding architecture having multi-head attention mechanisms.

### Claim 7 (depends on 3)

The computer-implemented method of claim 3, wherein the large language model comprises a Transformer-based large language model.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the response to the query comprises a textual response conveying the requested information as a coherent, focused response to the query.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein:
the query comprises a question in a natural language; and
the response to the query comprises a natural language response to the question.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the response comprises a summary of the image data based on the creator-provided text.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving image data comprising an image frame;
processing the image data to generate a structured document, the structured document comprising creator-provided text recognized in the image frame;
receiving a query from a user requesting information contained in the image data; and
processing the query and the structured document to generate a response to the query, the response conveying the requested information contained in the image data.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise providing, for output from a user device associated with user, the response to the query.

### Claim 13 (depends on 11)

The system of claim 11, wherein processing the query and the structured document to generate the response to the query comprises using a large language model to process the query and the structured document to generate the response to the query as output from the large language model.

### Claim 14 (depends on 13)

The system of claim 13, wherein the large language model comprises a pre-trained large language model.

### Claim 15 (depends on 14)

The system of claim 14, wherein the pre-trained large language model generates the response to the query by performing few-shot learning using the structured document as context for the query.

### Claim 16 (depends on 13)

The system of claim 13, wherein the large language model comprises an encoding/decoding architecture having multi-head attention mechanisms.

### Claim 17 (depends on 13)

The system of claim 13, wherein the large language model comprises a Transformer-based large language model.

### Claim 18 (depends on 11)

The system of claim 11, wherein the response to the query comprises a textual response conveying the requested information as a coherent, focused response to the query.

### Claim 19 (depends on 11)

The system of claim 11, wherein:
the query comprises a question in a natural language; and
the response to the query comprises a natural language response to the question.

### Claim 20 (depends on 11)

The system of claim 11, wherein the response comprises a summary of the image data based on the creator-provided text.
