US 12,217,494 B2Granted

Keyframe extractor

AssigneeGracenote, Inc.

InventorAndreas Schmidt

Life of this patent

Estimating the term…

Loading patent…

Abstract

In one aspect, an example method includes (i) determining a blur delta that quantifies a difference between a level of blurriness of a first frame of a video and a level of blurriness of a second frame of the video, wherein the second frame is subsequent to and adjacent to the first frame; (ii) determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame; (iii) determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame; (iv) determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance; (v) based on the keyframe score, determining that the second frame is a keyframe; and (vi) outputting data indicating that the second frame is a keyframe.

Claims17

Dependent claims

A method comprising:

determining, by a computing system, a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by:
converting the first frame to grayscale;
downscaling the grayscale first frame;
calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients;
transposing the first matrix of DCT coefficients;
calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients;
and
calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients;
determining, by the computing system, a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by:
converting the second frame to grayscale;
downscaling the grayscale second frame;
calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients;
transposing the third matrix of DCT coefficients;
calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients;
and
calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients;
determining, by the computing system, a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the second blur score, wherein the second frame is subsequent to and adjacent to the first frame;
determining, by the computing system, a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame;
determining, by the computing system, a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame;
determining, by the computing system, a keyframe score using the blur delta, the contrast delta, and the fingerprint distance;
based on the keyframe score, determining, by the computing system, that the second frame is a keyframe;
and
outputting, by the computing system, data indicating that the second frame is a keyframe.

20 more elements

A non-transitory computer-readable medium having stored thereon program instructions that upon execution by a processor, cause performance of a set of acts comprising:

determining a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by:
converting the first frame to grayscale;
downscaling the grayscale first frame;
calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients;
transposing the first matrix of DCT coefficients;
calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients;
and
calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients;
determining a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by:
converting the second frame to grayscale;
downscaling the grayscale second frame;
calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients;
transposing the third matrix of DCT coefficients;
calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients;
and
calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients;
determining a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the second blur score, wherein the second frame is subsequent to and adjacent to the first frame;
determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame;
determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame;
determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance;
based on the keyframe score, determining that the second frame is a keyframe;
and
outputting data indicating that the second frame is a keyframe.

20 more elements

A computing system comprising:

a processor;
and
a non-transitory computer-readable medium, having stored thereon program instructions that upon execution by the processor, cause the computing system to perform a set of acts comprising:
determining a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by:
converting the first frame to grayscale;
downscaling the grayscale first frame;
calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients;
transposing the first matrix of DCT coefficients;
calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients;
and
calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients;
determining a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by:
converting the second frame to grayscale;
downscaling the grayscale second frame;
calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients;
transposing the third matrix of DCT coefficients;
calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients;
and
calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients;
determining a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the first blur score, wherein the second frame is subsequent to and adjacent to the first frame;
determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame;
determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame;
determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance;
based on the keyframe score, determining that the second frame is a keyframe;
and
outputting data indicating that the second frame is a keyframe.

23 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/224,034Filed 6 Apr 2021 · us provisional application
Priority: 10 Apr 2020
Published: US 2021/0319230 A114 Oct 2021
Granted: 4 Feb 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification22 codes

G06V 10/82
Using pattern recognition or machine learning: using neural networks
G06F 18/24
Analysing: classification techniques
G06F 18/2413
Relating to the classification model, e.g. parametric or non-parametric approaches: based on distances to training or reference patterns
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks

Family

Application63/008,477Us provisional application · filed 10 Apr 2020
US 2021/0319230 A1Pre-grant publication · published 14 Oct 2021
US 12,217,494 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key