# US 2025/0097549 A1 · Multimedia scene break detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0097549 A1
- Document ID: US20250097549A1
- Status: Published application
- Kind: A1
- Application number: 18/964,224
- Filed: 2024-11-29
- Published: 2025-03-20
- Earliest claimed: 2023-04-17
- Assignees: Tubi, Inc.
- Applicants: Tubi, Inc.
- Inventors: Amir Mazaheri; John Trenkle; Jaya Kawale; Kevin Edward Corcoran; Dennis Paul Yost; Anthony Albert Truyoo-Broque; Matthew Adam Elliott
- Web page: https://patentgrep.com/patent/US20250097549A1

## CPC

- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G11B 27/19: by using information detectable on the record carrier
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A system and method for multimedia scene break detection, including: a computer processor, a scene break detection service executing on the computer processor and comprising functionality to: (i) receive a request for scene break detection on a media item, (ii) identify a set of candidate scene break timestamps selected based on analysis of an audio component of the media item and a video component of the media item, (iii) execute a computer vision scoring model for each candidate scene break timestamp to generate a score representing a visual distance between a first set of proximal frames and a second set of proximal frames of the candidate scene break timestamp, and (iv) select, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for the media item.

## Claims

### Claim 1

A system for computer vision analysis of a media item, comprising:
a computer processor;
a scene break detection service executing on the computer processor and comprising functionality to:
receive a request for scene break detection on the media item;
identify a set of candidate scene break timestamps selected based on analysis of an audio component of the media item and a video component of the media item;
execute a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first set of proximal frames preceding the candidate scene break timestamp and a second set of proximal frames succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first set of proximal frames and the second set of proximal frames; and
select, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for the media item.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first set of proximal frames are contiguous frames of a preceding shot, and wherein the second set of proximal frames are contiguous frames of a succeeding shot.

### Claim 3 (depends on 1)

The system of claim 1, wherein the system further comprises a media preview generation service configured to:
obtain a request for media preview generation for the media item;
identify a duration corresponding to the request for media preview generation; and
select, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as a media preview clip.

### Claim 4 (depends on 3)

The system of claim 3, wherein the two timestamps of the final set of scene break timestamps are adjacent, and wherein the media preview clip is provided for display in relation to the media item to one or more end users by a client device.

### Claim 5 (depends on 3)

The system of claim 3, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for media preview generation;
providing the set of candidate segments for human selection;
receiving a selection of a subset of the set of candidate segments obtained from a human administrator; and
constructing the media preview clip using the subset of the set of candidate segments based on the selection.

### Claim 6 (depends on 3)

The system of claim 3, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for media preview generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors each depicting at least one visual attribute of the candidate segment;
executing a machine learning model to infer a visual category for the candidate segment, wherein the visual category is selected from a set of visual categories;
selecting a contiguous subset of the set of candidate segments having a common inferred visual category; and
constructing the media preview clip using the contiguous subset of the set of candidate segments.

### Claim 7 (depends on 1)

The system of claim 1, wherein the system further comprises media preview generation service configured to:
obtain a request for advertising media clip generation for the media item;
identify a duration corresponding to the request; and
select, based on the duration, a segment of the media item between two timestamps of the final set of scene break timestamps as an advertising media clip.

### Claim 8 (depends on 7)

The system of claim 7, wherein selecting the segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for advertising media clip generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors each comprising at least one visual attribute of the candidate segment, a set of actor attributes, and a genre attribute;
executing, based on the set of feature vectors, a machine learning model to infer a virality score for the candidate segment;
determining that the segment has a highest virality score among the set of candidate segments; and
constructing the advertising media clip using the segment, wherein the advertising media clip is provided for display in an advertisement for the media item.

### Claim 9 (depends on 1)

The system of claim 1, wherein the scene break detection service further comprises functionality to:
perform shot detection on the video component of the media item to obtain a set of shots, each comprising a set of contiguous frames of the media item;
select the set of candidate scene break timestamps by comparing shot boundaries of the set of shots with a set of audio break timestamps identified based on aurally similar segments of the audio component.

### Claim 10 (depends on 1)

The system of claim 1, wherein executing the computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps further comprises:
constructing a graph representation of the first set of proximal frames and the second set of proximal frames; and
wherein calculating a score for the candidate scene break timestamp comprises performing bipartite matching on the graph representation to generate a set of distance values, each of the set of distance values representing a proximal distance between at least one of the first set of proximal frames and at least one of the second set of proximal frames, wherein the score for each of the candidate scene break timestamps is an average of the set of distance values.

### Claim 11 (depends on 1)

The system of claim 1, wherein selecting the final set of scene break timestamps comprises:
identifying a set of dynamic programming heuristic rules for scene break selection;
using the set of dynamic programming heuristic rules to generate a ranking of at least a subset of the set of candidate scene break timestamps; and
selecting the final set of scene break timestamps based on the ranking.

### Claim 12

A method for computer vision analysis of a media item, comprising:
receiving a request for scene break detection on the media item;
identifying a set of candidate scene break timestamps selected based on analysis of an audio component of the media item and a video component of the media item;
executing, by a computer processor, a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first set of proximal frames preceding the candidate scene break timestamp and a second set of proximal frames succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first set of proximal frames and the second set of proximal frames; and
selecting, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for the media item.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first set of proximal frames are contiguous frames of a preceding shot, and wherein the second set of proximal frames are contiguous frames of a succeeding shot.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
obtaining a request for media preview generation for the media item;
identifying a duration corresponding to the request for media preview generation; and
selecting, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as a media preview clip.

### Claim 15 (depends on 14)

The method of claim 14, wherein the two timestamps of the final set of scene break timestamps are adjacent, and wherein the media preview clip is provided for display in relation to the media item to one or more end users by a client device.

### Claim 16 (depends on 12)

The method of claim 12, further comprising:
obtaining a request for advertising media clip generation for the media item;
identifying a duration corresponding to the request; and
selecting, based on the duration, a segment of the media item between two timestamps of the final set of scene break timestamps as an advertising media clip.

### Claim 17 (depends on 16)

The method of claim 16, wherein selecting the segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for advertising media clip generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors each comprising at least one visual attribute of the candidate segment, a set of actor attributes, and a genre attribute;
executing, based on the set of feature vectors, a machine learning model to infer a virality score for the candidate segment;
determining that the segment has a highest virality score among the set of candidate segments; and
constructing the advertising media clip using the segment, wherein the advertising media clip is provided for display in an advertisement for the media item.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
performing shot detection on the video component of the media item to obtain a set of shots, each comprising a set of contiguous frames of the media item;
selecting the set of candidate scene break timestamps by comparing shot boundaries of the set of shots with a set of audio break timestamps identified based on aurally similar segments of the audio component.

### Claim 19 (depends on 12)

The method of claim 12, wherein executing the computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps further comprises:
constructing a graph representation of the first set of proximal frames and the second set of proximal frames; and
wherein calculating a score for the candidate scene break timestamp comprises performing bipartite matching on the graph representation to generate a set of distance values, each of the set of distance values representing a proximal distance between at least one of the first set of proximal frames and at least one of the second set of proximal frames, wherein the score for each of the candidate scene break timestamps is an average of the set of distance values.

### Claim 20

A non-transitory computer-readable storage medium comprising a plurality of instructions for computer vision analysis of a media item, the plurality of instructions configured to execute on at least one computer processor to enable the at least one computer processor to:
receive a request for scene break detection on the media item;
identify a set of candidate scene break timestamps selected based on analysis of an audio component of the media item and a video component of the media item;
execute a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first set of proximal frames preceding the candidate scene break timestamp and a second set of proximal frames succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first set of proximal frames and the second set of proximal frames; and
select, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for the media item.
