# US 2024/0406518 A1 · Tracking content with artifical intelligence as it is consumed

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0406518 A1
- Document ID: US20240406518A1
- Status: Published application
- Kind: A1
- Application number: 18/328,533
- Filed: 2023-06-02
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Brant Candelore
- Web page: https://patentgrep.com/patent/US20240406518A1

## CPC

- H04N 21/42209: {for displaying non-command information, e.g. electronic programme guide [EPG], e-mail, messages or a second television channel}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/4828: {for searching programme descriptors (retrieval of video data G06F16/739)}
- H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection
- H04N 21/845: Structuring of content, e.g. decomposing content into time segments
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A user starts playback of an audio or AV program, which starts up a large language model (LLM) such as a generative pre-trained transformer. The LLM follows the program with the user as the program is being watched. Audio is converted to text and video is converted to text-based description. The LLM essentially trains on the resulting “corpus” so that should the user subsequently want to access part of the program using a simple conversational query about the program, the LLM can provide the answer.

## Claims

### Claim 1

An apparatus, comprising:
at least one computer memory that is not a transitory signal and that comprises instructions executable by at least one processor assembly to:
receive a command to present audio and/or audio video (AV) content on a display;
responsive to the command, start to play the content;
responsive to the command, start at least one machine learning (ML) model to receive audio and/or video being played in the content;
use the audio and/or video in the content to train at least one large language model (LLM);
receive at least one query related to the content;
input the query to the LLM;
receive from ML model a response to the query; and
present the response on at least one output device.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the ML model comprises at least one generative pre-trained transformer.

### Claim 3 (depends on 1)

The apparatus of claim 1, comprising the at least one processor assembly.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the instructions are executable to:
use the audio and/or video in the content to train the LLM at least in part by converting at least one of: the audio, the video, to text and inputting the text to the LLM.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the content is played by a content player and the instructions are executable to:
provide indication to the LLM of at least a first command input to the content player.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the LLM is executed by the content player.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein the first command is a stop or pause command.

### Claim 8 (depends on 5)

The apparatus of claim 5, wherein the first command is a trick play command.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the ML model is separate from the LLM.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the LLM comprises the ML model.

### Claim 11

A method, comprising:
playing content using a content player;
as the content is being played, using the content to train at least one machine learning (ML) model; and
using the ML model to return at least one response to at least one query about the content.

### Claim 12 (depends on 11)

The method of claim 11, comprising using only portions of the content that have been played to train the ML model to return the response to the query.

### Claim 13 (depends on 11)

The method of claim 11, comprising using more than portions of the content that have been played to train the ML model to return the response to the query.

### Claim 14 (depends on 11)

The method of claim 11, comprising generating text from the content and using the text to train the ML model.

### Claim 15

An apparatus comprising:
at least one processor assembly configured to:
invoke at least one machine learning (ML) model to consume information related to content as the content is being played on at least one output device to train the ML model; and
use the ML model to respond to at least one query about the content.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the ML model comprises at least one large language model (LLM).

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the LLM comprises at least one generative pre-trained transformer.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein the processor assembly is configured to generate text from the content to train the ML model on the text.

### Claim 19 (depends on 15)

The apparatus of claim 15, wherein the processor assembly is configured to train the ML model only on portions of the content that have been played to train the ML model to respond to the query.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein the processor assembly is configured to train the ML model on more than portions of the content that have been played to train the ML model to respond to the query.
