# US 12,217,494 B2 · Keyframe extractor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,494 B2 - Document ID: US12217494B2 - Status: Granted - Kind: B2 - Application number: 17/224,034 - Filed: 2021-04-06 - Published: 2021-10-14 - Granted: 2025-02-04 - Earliest claimed: 2020-04-10 - Assignees: Gracenote, Inc. - Applicants: Gracenote, Inc. - Inventors: Andreas Schmidt - Web page: https://patentgrep.com/patent/US12217494B2 ## CPC - G06V 10/82: using neural networks - G06F 18/24: Classification techniques - G06F 18/2413: based on distances to training or reference patterns - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs - G06N 3/08: Learning methods - G06V 10/806: {of extracted features} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - H04N 21/2547: Third Party Billing, e.g. billing of advertiser - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} - H04N 5/147: {Scene change detection} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract In one aspect, an example method includes (i) determining a blur delta that quantifies a difference between a level of blurriness of a first frame of a video and a level of blurriness of a second frame of the video, wherein the second frame is subsequent to and adjacent to the first frame; (ii) determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame; (iii) determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame; (iv) determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance; (v) based on the keyframe score, determining that the second frame is a keyframe; and (vi) outputting data indicating that the second frame is a keyframe. ## Claims ### Claim 1 A method comprising: determining, by a computing system, a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by: converting the first frame to grayscale; downscaling the grayscale first frame; calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients; transposing the first matrix of DCT coefficients; calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients; and calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients; determining, by the computing system, a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by: converting the second frame to grayscale; downscaling the grayscale second frame; calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients; transposing the third matrix of DCT coefficients; calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients; and calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients; determining, by the computing system, a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the second blur score, wherein the second frame is subsequent to and adjacent to the first frame; determining, by the computing system, a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame; determining, by the computing system, a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame; determining, by the computing system, a keyframe score using the blur delta, the contrast delta, and the fingerprint distance; based on the keyframe score, determining, by the computing system, that the second frame is a keyframe; and outputting, by the computing system, data indicating that the second frame is a keyframe. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the contrast delta comprises: determining a contrast score for the first frame based on a standard deviation of a histogram of pixel intensity values of the first frame; and determining a contrast score for the second frame based on a standard deviation of a histogram of pixel intensity values of the second frame. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining, based on the contrast score for the second frame, that the second frame is a blackframe; and outputting data indicating that the second frame is a blackframe. ### Claim 4 (depends on 1) The method of claim 1, wherein: the first image fingerprint is based on features extracted from a set of regions of the first frame; and the second image fingerprint is based on features extracted from a set of regions of the second frame. ### Claim 5 (depends on 1) The method of claim 1, wherein determining that the second frame is a keyframe comprises determining that the keyframe score satisfies a threshold condition. ### Claim 6 (depends on 1) The method of claim 1, further comprising using the data indicating that the second frame is a keyframe to refine transition data output by a transition detection classifier, wherein the transition data is indicative of locations within the video of transitions between advertisement content and program content. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying a third frame of the video as a keyframe based on a keyframe score for the third frame and a fourth frame of the video that is prior to and adjacent to the third frame; identifying a segment of the video between the second frame and the fourth frame as a query segment; and searching for a match to the query segment within a video database. ### Claim 8 A non-transitory computer-readable medium having stored thereon program instructions that upon execution by a processor, cause performance of a set of acts comprising: determining a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by: converting the first frame to grayscale; downscaling the grayscale first frame; calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients; transposing the first matrix of DCT coefficients; calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients; and calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients; determining a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by: converting the second frame to grayscale; downscaling the grayscale second frame; calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients; transposing the third matrix of DCT coefficients; calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients; and calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients; determining a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the second blur score, wherein the second frame is subsequent to and adjacent to the first frame; determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame; determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame; determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance; based on the keyframe score, determining that the second frame is a keyframe; and outputting data indicating that the second frame is a keyframe. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein determining the contrast delta comprises: determining a contrast score for the first frame based on a standard deviation of a histogram of pixel intensity values of the first frame; and determining a contrast score for the second frame based on a standard deviation of a histogram of pixel intensity values of the second frame. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the set of acts further comprises: determining, based on the contrast score for the second frame, that the second frame is a blackframe; and outputting data indicating that the second frame is a blackframe. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein: the first image fingerprint is based on features extracted from a set of regions of the first frame; and the second image fingerprint is based on features extracted from a set of regions of the second frame. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the set of acts further comprises using the data indicating that the second frame is a keyframe to refine transition data output by a transition detection classifier, wherein the transition data is indicative of locations within the video of transitions between advertisement content and program content. ### Claim 13 A computing system comprising: a processor; and a non-transitory computer-readable medium, having stored thereon program instructions that upon execution by the processor, cause the computing system to perform a set of acts comprising: determining a first blur score for a first frame of a video based on a discrete cosine transform (DCT) of pixel intensity values of the first frame by: converting the first frame to grayscale; downscaling the grayscale first frame; calculating a first DCT of the grayscale, downscaled first frame to determine a first matrix of DCT coefficients; transposing the first matrix of DCT coefficients; calculating a second DCT of the transposed first matrix of DCT coefficients to determine a second matrix of DCT coefficients; and calculating the first blur score for the first frame by subtracting a sum of the absolute values of the upper-left quarter of the second matrix of DCT coefficients from a sum of the absolute values of the coefficients of the second matrix of DCT coefficients; determining a second blur score for a second frame of the video based on a DCT of pixel intensity values of the second frame by: converting the second frame to grayscale; downscaling the grayscale second frame; calculating a third DCT of the grayscale, downscaled second frame to determine a third matrix of DCT coefficients; transposing the third matrix of DCT coefficients; calculating a fourth DCT of the transposed third matrix of DCT coefficients to determine a fourth matrix of DCT coefficients; and calculating the second blur score for the second frame by subtracting a sum of the absolute values of the upper-left quarter of the fourth matrix of DCT coefficients from a sum of the absolute values of the coefficients of the fourth matrix of DCT coefficients; determining a blur delta that quantifies a difference between a level of blurriness of the first frame of the video represented by the first blur score and a level of blurriness of the second frame of the video represented by the first blur score, wherein the second frame is subsequent to and adjacent to the first frame; determining a contrast delta that quantifies a difference between a contrast of the first frame and a contrast of the second frame; determining a fingerprint distance between a first image fingerprint of the first frame and a second image fingerprint of the second frame; determining a keyframe score using the blur delta, the contrast delta, and the fingerprint distance; based on the keyframe score, determining that the second frame is a keyframe; and outputting data indicating that the second frame is a keyframe. ### Claim 14 (depends on 13) The computing system of claim 13, wherein determining the contrast delta comprises: determining a contrast score for the first frame based on a standard deviation of a histogram of pixel intensity values of the first frame; and determining a contrast score for the second frame based on a standard deviation of a histogram of pixel intensity values of the second frame. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the set of acts further comprises: determining, based on the contrast score for the second frame, that the second frame is a blackframe; and outputting data indicating that the second frame is a blackframe. ### Claim 16 (depends on 13) The computing system of claim 13, wherein: the first image fingerprint is based on features extracted from a set of regions of the first frame; and the second image fingerprint is based on features extracted from a set of regions of the second frame. ### Claim 17 (depends on 13) The computing system of claim 13, wherein the set of acts further comprises using the data indicating that the second frame is a keyframe to refine transition data output by a transition detection classifier, wherein the transition data is indicative of locations within the video of transitions between advertisement content and program content.