# US 2024/0404285 A1 · Unsupervised cue point discovery for episodic content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0404285 A1
- Document ID: US20240404285A1
- Status: Published application
- Kind: A1
- Application number: 18/205,802
- Filed: 2023-06-05
- Published: 2024-12-05
- Earliest claimed: 2023-06-05
- Assignees: Roku, Inc.
- Applicants: Roku, Inc.
- Inventors: MICHAEL CUTTER; ROHIT NYAYAPATI; SUNIL RAMESH
- Web page: https://patentgrep.com/patent/US20240404285A1

## CPC

- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/63: {Scene text, e.g. street names}
- G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques}
- H04N 21/8455: {involving pointers to the content, e.g. pointers to the I-frames of the video stream}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for cue point discovery for content. For example, system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof are provided for using unsupervised machine learning to automatically classify cue points for episodic content. The cue points can be associates with an opening credits section, an end credits section, a recap section, or a behind-the-scenes section.

## Claims

### Claim 1

A computer-implemented method, comprising:
dividing, by at least one computer processor, a video associated with an episode of an episodic content into a plurality of sections;
determining a representation for each of the plurality of sections;
comparing a first representation associated with a first section of the plurality of sections of the video with a plurality of representations, wherein the plurality of representations are associated with one or more sections of one or more episodes of the episodic content;
determining a plurality of similarity values for the first representation based on the comparison;
determining one or more of the plurality of similarity values that satisfy a condition;
determining a temporal position associated with the first representation in response to the one or more of the plurality of similarity values satisfying the condition; and
using the temporal position associated with the first representation to determine a type of the first section of the plurality of sections of the video.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the representation comprises an image embedding, an audio embedding, a text embedding, or a combination of two or more of the image embedding, the audio embedding, and the text embedding.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the type of the first section comprises an opening credits section, an end credits section, a recap section, or a behind-the-scenes section.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein using the temporal position associated with the first representation to determine the type of the first section comprises:
comparing the temporal position with a first temporal position threshold; and
determining that the type of the first section comprises the opening credits section in response to the temporal position being before the first temporal position threshold.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein using the temporal position associated with the first representation to determine the type of the first section comprises:
comparing the temporal position with a first temporal position threshold; and
determining that the type of the first section comprises the end credits section in response to the temporal position being after the first temporal position threshold.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein determining the type of the first section further comprises:
using a text detection method to determine text within the first section of the plurality of sections of the video; and
using the determined text to determine that the type of the first section comprises the end credits section.

### Claim 7 (depends on 3)

The computer-implemented method of claim 3, wherein determining the type of the first section further comprises:
using at least one or more of first region information associated with a first region where the episodic content is produced, second region information associated with a second region where the episodic content is being shown, or production information associated with the episodic content to determine the type of the first section.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein determining one or more of the plurality of similarity values that satisfy the condition comprises:
comparing the plurality of similarity values with a threshold; and
determining that the one or more of the plurality of similarity values are greater than the threshold.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein determining the representation, comparing the first representation with the plurality of representations, determining the plurality of similarity values, and determining the one or more of the plurality of similarity values satisfying the condition are part of an unsupervised machine learning model.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining two or more sections of the plurality of sections of the video that have similarity values that satisfy the condition;
determining a number of the two or more sections; and
in response to the number of the two or more sections satisfying a second threshold, using temporal positions associated with the two or more sections to determine a type of the two or more sections of the plurality of sections of the video.

### Claim 11

A system, comprising:
one or more memories; and
at least one processor each coupled to at least one of the memories and configured to perform operations comprising:
dividing a video associated with an episode of an episodic content into a plurality of sections;
determining a representation for each of the plurality of sections;
comparing a first representation associated with a first section of the plurality of sections of the video with a plurality of representations, wherein the plurality of representations are associated with one or more sections of one or more episodes of the episodic content;
determining a plurality of similarity values for the first representation based on the comparison;
determining one or more of the plurality of similarity values that satisfy a condition;
determining a temporal position associated with the first representation in response to the one or more of the plurality of similarity values satisfying the condition; and
using the temporal position associated with the first representation to determine a type of the first section of the plurality of sections of the video.

### Claim 12 (depends on 11)

The system of claim 11, wherein:
the representation comprises an image embedding, an audio embedding, a text embedding, or a combination of two or more of the image embedding, the audio embedding, and the text embedding; and
the type of the first section comprises an opening credits section, an end credits section, a recap section, or a behind-the-scenes section.

### Claim 13 (depends on 12)

The system of claim 12, wherein using the temporal position associated with the first representation to determine the type of the first section comprises:
comparing the temporal position with a first temporal position threshold; and
determining that the type of the first section comprises the opening credits section in response to the temporal position being before the first temporal position threshold.

### Claim 14 (depends on 12)

The system of claim 12, wherein using the temporal position associated with the first representation to determine the type of the first section comprises:
comparing the temporal position with a first temporal position threshold; and
determining that the type of the first section comprises the end credits section in response to the temporal position being after the first temporal position threshold.

### Claim 15 (depends on 12)

The system of claim 12, wherein determining the type of the first section further comprises:
using a text detection method to determine text within the first section of the plurality of sections of the video; and
using the determined text to determine that the type of the first section comprises the end credits section.

### Claim 16 (depends on 12)

The system of claim 12, wherein determining the type of the first section further comprises:
using at least one or more of first region information associated with a first region where the episodic content is produced, second region information associated with a second region where the episodic content is being shown, or production information associated with the episodic content to determine the type of the first section.

### Claim 17 (depends on 11)

The system of claim 11, wherein determining one or more of the plurality of similarity values that satisfy the condition comprises:
comparing the plurality of similarity values with a threshold; and
determining that the one or more of the plurality of similarity values are greater than the threshold.

### Claim 18 (depends on 11)

The system of claim 11, wherein determining the representation, comparing the first representation with the plurality of representations, determining the plurality of similarity values, and determining the one or more of the plurality of similarity values satisfying the condition are part of an unsupervised machine learning model.

### Claim 19

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
dividing a video associated with an episode of an episodic content into a plurality of sections;
determining a representation for each of the plurality of sections;
comparing a first representation associated with a first section of the plurality of sections of the video with a plurality of representations, wherein the plurality of representations are associated with one or more sections of one or more episodes of the episodic content;
determining a plurality of similarity values for the first representation based on the comparison;
determining one or more of the plurality of similarity values that satisfy a condition;
determining a temporal position associated with the first representation in response to the one or more of the plurality of similarity values satisfying the condition; and
using the temporal position associated with the first representation to determine a type of the first section of the plurality of sections of the video, wherein the type of the first section comprises an opening credits section, an end credits section, a recap section, or a behind-the-scenes section.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein using the temporal position associated with the first representation to determine the type of the first section comprises:
comparing the temporal position with a first temporal position threshold and a second temporal position threshold;
determining that the type of the first section comprises the opening credits section in response to the temporal position being before the first temporal position threshold; and
determining that the type of the first section comprises the end credits section in response to the temporal position being after the second temporal position threshold.
