# US 2026/0267916 A1 · Structured video documents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0267916 A1
- Document ID: US20260267916A1
- Status: Published application
- Kind: A1
- Application number: 19/654,023
- Filed: 2026-04-21
- Published: 2026-09-10
- Earliest claimed: 2022-03-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Johan Schalkwyk; Françoise Beaufays
- Web page: https://patentgrep.com/patent/US20260267916A1

## CPC

- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G10L 15/04: Segmentation; Word boundary detection
- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 16/7834: {using audio features}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/30: Semantic analysis
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method includes receiving a content feed that includes audio data corresponding to speech utterances and processing the content feed to generate a semantically-rich, structured document. The structured document includes a transcription of the speech utterances and includes a plurality of words each aligned with a corresponding audio segment of the audio data that indicates a time when the word was recognized in the audio data. During playback of the content feed, the method also includes receiving a query from a user requesting information contained in the content feed and processing, by a large language model, the query and the structured document to generate a response to the query. The response conveys the requested information contained in the content feed. The method also includes providing, for output from a user device associated with the user, the response to the query.

## Claims

### Claim 1

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a content feed comprising audio data and image data, the audio data corresponding to speech utterances;
processing the content feed to generate a semantically-rich, structured document, wherein the structured document comprises:
a transcription of the speech utterances, the transcription comprising a plurality of words each aligned with a corresponding audio segment of the audio data; and
creator-provided text recognized in one or more image frames of the image data via optical character recognition;
grouping the transcription and the creator-provided text into one or more sections of the structured document based on the corresponding audio segments to consolidate content corresponding to a respective time in the content feed;
receiving a query from a user requesting information contained in the content feed; and
processing, using a large language model, the natural language query and the semantically-rich, structured document as context to generate a response to the query.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise providing, for output from a user device associated with user, the response to the query.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the large language model comprises a pre-trained large language model.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the pre-trained large language model generates the response to the query by performing few-shot learning using the semantically, rich structured document as context for the query.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the large language model comprises an encoding/decoding architecture having multi-head attention mechanisms.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the large language model comprises a Transformer-based large language model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the response to the query comprises a textual response conveying the requested information as a coherent, focused response to the query.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein:
the natural language query comprises a question in a natural language; and
the response to the query comprises a natural language response to the question.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein processing the natural language query and the structured document comprises performing, by the large language model, a generative task to generate natural language text that summarizes one or more portions of the structured document.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the response comprises a summary of the content feed.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a content feed comprising audio data and image data, the audio data corresponding to speech utterances;
processing the content feed to generate a semantically-rich, structured document, wherein the structured document comprises:
a transcription of the speech utterances, the transcription comprising a plurality of words each aligned with a corresponding audio segment of the audio data; and
creator-provided text recognized in one or more image frames of the image data via optical character recognition,
grouping the transcription and the creator-provided text into one or more sections of the structured document based on the corresponding audio segments to consolidate content corresponding to a respective time in the content feed;
receiving a query from a user requesting information contained in the content feed; and
processing, using a large language model, the natural language query and the semantically-rich, structured document as context to generate a response to the query.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise providing, for output from a user device associated with user, the response to the query.

### Claim 13 (depends on 11)

The system of claim 11, wherein the large language model comprises a pre-trained large language model.

### Claim 14 (depends on 13)

The system of claim 13, wherein the pre-trained large language model generates the response to the query by performing few-shot learning using the semantically, rich structured document as context for the query.

### Claim 15 (depends on 11)

The system of claim 11, wherein the large language model comprises an encoding/decoding architecture having multi-head attention mechanisms.

### Claim 16 (depends on 11)

The system of claim 11, wherein the large language model comprises a Transformer-based large language model.

### Claim 17 (depends on 11)

The system of claim 11, wherein the response to the query comprises a textual response conveying the requested information as a coherent, focused response to the query.

### Claim 18 (depends on 11)

The system of claim 11, wherein:
the natural language query comprises a question in a natural language; and
the response to the query comprises a natural language response to the question.

### Claim 19 (depends on 11)

The system of claim 11, wherein processing the natural language query and the structured document comprises performing, by the large language model, a generative task to generate natural language text that summarizes one or more portions of the structured document.

### Claim 20 (depends on 19)

The system of claim 19, wherein the response comprises a summary of the content feed.
