# US 10,503,956 B2 · Identifying video content via color-based fingerprint matching > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,956 B2 - Document ID: US10503956B2 - Status: Granted - Kind: B2 - Application number: 15/286,787 - Filed: 2016-10-06 - Published: 2017-03-30 - Granted: 2019-12-10 - Earliest claimed: 2013-10-23 - Assignees: Gracenote, Inc. - Applicants: Gracenote, Inc. - Inventors: Wilson Harron; Markus K. Cremer - Web page: https://patentgrep.com/patent/US10503956B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 16/783: using metadata automatically derived from the content - G06V 10/56: relating to colour - G06V 20/48: {Matching video sequences} - G06V 40/1365: {Matching; Classification} - G06V 40/50: Maintenance of biometric data or enrolment thereof - G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods of identifying unknown video content are described. An example method may include receiving a first fingerprint and a second fingerprint. The first fingerprint may be a color-based fingerprint derived from colors in a portion of the unknown video content, and the second fingerprint may be at least partially based on a feature other than the colors of the unknown video content. A reference database of reference fingerprints may then be queried using one of the first fingerprint or the second fingerprint to obtain a candidate group of fingerprints. The candiate group of fingerprints may then be pried using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint. The unknown video content may then be identified using the at least one query fingerprint. For example, the second fingerprint may be a luminance-based fingerprint derived from luminance in the unknown video content. ## Claims ### Claim 1 A method comprising: processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 2 (depends on 1) The method of claim 1, further comprising receiving the unknown video content at a media playback device including a hardware processor, the query of the at least one reference database and the query of the candidate group of fingerprints being performed by the hardware processor. ### Claim 3 (depends on 2) The method of claim 2, wherein the at least one reference database is stored in memory of the media playback device. ### Claim 4 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database of reference fingerprints comprises sending the first fingerprint and the second fingerprint to a remote server system to execute the query, the at least one reference database being part of the remote server system. ### Claim 5 (depends on 1) The method of claim 1, wherein the at least one reference database comprises: a color-based fingerprint database including reference fingerprints that are color-based; and a patch-based fingerprint database including reference fingerprints that are patch-based. ### Claim 6 (depends on 1) The method of claim 1, wherein the at least one reference database is queried with the first fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the second fingerprint to identify the at least one query fingerprint. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint. ### Claim 8 (depends on 1) The method of claim 1, wherein the portion of the unknown video content corresponds to at least one frame of the unknown video content. ### Claim 9 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database using one of the first fingerprint or the second fingerprint uses a fingerprint defined by a lower number of bits than that used to query the candidate group. ### Claim 10 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database comprises querying the at least one reference database of reference fingerprints using the color-based fingerprint. ### Claim 11 (depends on 10) The method of claim 10, wherein querying the at least one reference database using the color-based fingerprint comprises determining that a similarity between the color-based fingerprint and the reference fingerprints satisfies a predetermined threshold associated with vector distance measurement. ### Claim 12 (depends on 1) The method of claim 1, wherein the patches are defined by regions of at least one frame of the unknown video content. ### Claim 13 A computer comprising: one or more hardware processors; and memory including instructions that, when executed by the one or more processors, cause the computer to perform operations comprising: processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 14 (depends on 13) The computer of claim 13, wherein the computer is a media playback device including the at least one reference database, the media playback device configured to receive the unknown video content and query the at least one reference database and the candidate group of fingerprints to identify the at least one query fingerprint. ### Claim 15 (depends on 13) The computer of claim 13, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint. ### Claim 16 A non-transitory computer-readable storage medium embodying instructions which, when executed by a machine, cause the machine to perform operations comprising: processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the machine is a media playback device, further comprising instructions to receive the unknown video content at the media playback device including a hardware processor, and query the at least one reference database and the candidate group of fingerprints using the hardware processor. ### Claim 18 (depends on 8) The method of claim 8, wherein the plurality of colors are the most dominant colors within the at least one frame. ### Claim 19 (depends on 8) The method of claim 8, wherein the color-based fingerprint includes 8 color-based values determined for the 8 most dominant colors within the at least one frame. ### Claim 20 (depends on 8) The method of claim 8, wherein the patch-based fingerprint includes 32 patch-based values determined for 32 patches in the at least one frame.