# US 12,192,600 B2 · Multimedia scene break detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,192,600 B2
- Document ID: US12192600B2
- Status: Granted
- Kind: B2
- Application number: 18/301,971
- Filed: 2023-04-17
- Published: 2024-10-24
- Granted: 2025-01-07
- Earliest claimed: 2023-04-17
- Assignees: Tubi, Inc.
- Applicants: Tubi, Inc.
- Inventors: Amir Mazaheri; John Trenkle; Jaya Kawale; Kevin Edward Corcoran; Dennis Paul Yost; Anthony Albert Truyoo-Broque; Matthew Adam Elliott
- Web page: https://patentgrep.com/patent/US12192600B2

## CPC

- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G11B 27/19: by using information detectable on the record carrier
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A system and method for multimedia scene break detection, including: a computer processor, a scene break detection service executing on the computer processor and comprising functionality to: (i) receive a request for scene break detection on the media item, (ii) perform audio break detection on an audio component of the media item to obtain a set of audio break timestamps, (iii) identify a set of video break timestamps, each corresponding to at least one frame of a video component of the media item, (iv) identify a set of candidate scene break timestamps based on the audio break timestamps and the video break timestamps, (v) execute a computer vision scoring model for each candidate scene break timestamp, and (vi) select, based at least on a score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for performing a media action.

## Claims

### Claim 1

A system for computer vision analysis of a media item, comprising:
a computer processor;
a scene break detection service executing on the computer processor and comprising functionality to:
receive a request for scene break detection on the media item;
perform audio break detection on an audio component of the media item to obtain a set of audio break timestamps corresponding to aurally similar segments of the audio component;
identify a set of video break timestamps, each corresponding to at least one frame of a video component of the media item;
identify a set of candidate scene break timestamps corresponding to instances of the set of the audio break timestamps and the set of video break timestamps within a predefined proximity;
execute a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first subset of the set of contiguous shots preceding the candidate scene break timestamp and a second subset of the set of contiguous shots succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first subset of contiguous shots and the second subset of contiguous shots; and
select, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for performing a media action.

### Claim 2 (depends on 1)

The system of claim 1, wherein the media action is media preview generation and wherein the system further comprises a media preview generation service configured to:
obtain a request for media preview generation for the media item;
identify a duration corresponding to the request; and
select, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as a media preview clip.

### Claim 3 (depends on 2)

The system of claim 2, wherein the two timestamps of the final set of scene break timestamps are adjacent, and wherein the media preview clip is provided for display in relation to the media item to one or more end users by a client device.

### Claim 4 (depends on 2)

The system of claim 2, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for media preview generation;
providing the set of candidate segments for human selection;
receiving a selection of a subset of the set of candidate segments obtained from a human administrator; and
constructing the media preview clip using the subset of the set of candidate segments based on the selection.

### Claim 5 (depends on 2)

The system of claim 2, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for media preview generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors depicting at least one visual attribute of the candidate segment;
executing a machine learning model to infer a visual category for the candidate segment, wherein the visual category is selected from a clustered set of visual categories;
selecting a subset of the set of candidate segments having a common inferred visual category; and
constructing the media preview clip using the subset of the set of candidate segments.

### Claim 6 (depends on 1)

The system of claim 1, wherein the media action is advertising media clip generation and wherein the system further comprises media preview generation service configured to:
obtain a request for advertising media clip generation for the media item;
identify a duration corresponding to the request; and
select, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as an advertising media clip.

### Claim 7 (depends on 6)

The system of claim 6, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for advertising media clip generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors comprising at least one visual attribute of the candidate segment, a set of actor attributes, and a genre attribute;
executing, based on the set of feature vectors, a machine learning model to infer a virality score for the candidate segment;
selecting a candidate segment of the set of candidate segments having a highest virality score; and
constructing the media preview clip using the candidate segment, wherein the media preview clip is provided for display in an advertisement for the media item.

### Claim 8 (depends on 1)

The system of claim 1, wherein the scene break detection service further comprises functionality to:
perform shot detection on the video component of the media item to obtain the set of video break timestamps corresponding to a set of shots of the video component, wherein each of the set of shots corresponds to a segment of the video component grouped at least partially on the basis of visual similarity.

### Claim 9 (depends on 1)

The system of claim 1, wherein executing the computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps further comprises:
constructing a graph representation of the first subset of contiguous shots and the second subset of contiguous shots; and
wherein calculating a score for the candidate scene break timestamp comprises performing bipartite matching on the graph representation to generate a set of distance values, each of the set of distance values representing a distance between at least one of the first subset of contiguous shots and at least one of the second subset of contiguous shots.

### Claim 10 (depends on 9)

The system of claim 9, wherein the score for each of the candidate scene break timestamps is an average of the set of distance values.

### Claim 11 (depends on 1)

The system of claim 1, wherein selecting the final set of scene break timestamps for performing the media action comprises:
identifying a set of dynamic programming heuristic rules for scene break selection;
using the set of dynamic programming heuristic rules to generate a ranking of the final set of ad break timestamps; and
selecting the final set of scene break timestamps based on the ranking.

### Claim 12

A method for computer vision analysis of a media item, comprising:
receiving a request for scene break detection on the media item;
performing audio break detection on an audio component of the media item to obtain a set of audio break timestamps corresponding to aurally similar segments of the audio component;
identifying a set of video break timestamps, each corresponding to at least one frame of a video component of the media item;
identifying a set of candidate scene break timestamps corresponding to instances of the set of the audio break timestamps and the set of video break timestamps within a predefined proximity;
executing, by a computer processor, a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first subset of the set of contiguous shots preceding the candidate scene break timestamp and a second subset of the set of contiguous shots succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first subset of contiguous shots and the second subset of contiguous shots; and
selecting, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for performing a media action.

### Claim 13 (depends on 12)

The method of claim 12, wherein the media action is media preview generation and wherein the method further comprises:
obtaining a request for media preview generation for the media item;
identifying a duration corresponding to the request; and
selecting, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as a media preview clip.

### Claim 14 (depends on 13)

The method of claim 13, wherein the two timestamps of the final set of scene break timestamps are adjacent, and wherein the media preview clip is provided for display in relation to the media item to one or more end users by a client device.

### Claim 15 (depends on 12)

The system of claim 12, wherein the media action is advertising media clip generation and wherein the method further comprises:
obtaining a request for advertising media clip generation for the media item;
identifying a duration corresponding to the request; and
selecting, based on the duration, at least one segment of the media item between two timestamps of the final set of scene break timestamps as an advertising media clip.

### Claim 16 (depends on 15)

The method of claim 15, wherein selecting the at least one segment of the media item comprises:
identifying a set of candidate segments of the media item based on the final set of scene break timestamps and a set of predefined criteria for advertising media clip generation;
for each candidate segment of the set of candidate segments:
generating a set of feature vectors comprising at least one visual attribute of the candidate segment, a set of actor attributes, and a genre attribute;
executing, based on the set of feature vectors, a machine learning model to infer a virality score for the candidate segment;
selecting a candidate segment of the set of candidate segments having a highest virality score; and
constructing the media preview clip using the candidate segment, wherein the media preview clip is provided for display in an advertisement for the media item.

### Claim 17 (depends on 12)

The method of claim 12, further comprising:
performing shot detection on the video component of the media item to obtain the set of video break timestamps corresponding to a set of shots of the video component, wherein each of the set of shots corresponds to a segment of the video component grouped at least partially on the basis of visual similarity.

### Claim 18 (depends on 12)

The method of claim 12, wherein executing the computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps further comprises:
constructing a graph representation of the first subset of contiguous shots and the second subset of contiguous shots; and
wherein calculating a score for the candidate scene break timestamp comprises performing bipartite matching on the graph representation to generate a set of distance values, each of the set of distance values representing a distance between at least one of the first subset of contiguous shots and at least one of the second subset of contiguous shots.

### Claim 19 (depends on 12)

The method of claim 12, wherein selecting the final set of scene break timestamps for performing the media action comprises:
identifying a set of dynamic programming heuristic rules for scene break selection;
using the set of dynamic programming heuristic rules to generate a ranking of the final set of ad break timestamps; and
selecting the final set of scene break timestamps based on the ranking.

### Claim 20

A non-transitory computer-readable storage medium comprising a plurality of instructions for computer vision analysis of a media item, the plurality of instructions configured to execute on at least one computer processor to enable the at least one computer processor to:
receive a request for scene break detection on the media item;
perform audio break detection on an audio component of the media item to obtain a set of audio break timestamps corresponding to aurally similar segments of the audio component;
identify a set of video break timestamps, each corresponding to at least one frame of a video component of the media item;
identify a set of candidate scene break timestamps corresponding to instances of the set of the audio break timestamps and the set of video break timestamps within a predefined proximity;
execute a computer vision scoring model for each candidate scene break timestamp of the set of candidate scene break timestamps by:
identifying a first subset of the set of contiguous shots preceding the candidate scene break timestamp and a second subset of the set of contiguous shots succeeding the candidate scene break timestamp;
calculating a score for the candidate scene break timestamp representing a visual distance between the first subset of contiguous shots and the second subset of contiguous shots; and
select, based at least on the score of each of the set of candidate scene break timestamps, a final set of scene break timestamps for performing a media action.