# US 11,308,731 B2 · Identifying video content via color-based fingerprint matching > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,731 B2 - Document ID: US11308731B2 - Status: Granted - Kind: B2 - Application number: 16/692,124 - Filed: 2019-11-22 - Published: 2020-03-19 - Granted: 2022-04-19 - Earliest claimed: 2013-10-23 - Assignees: Roku, Inc. - Applicants: Roku, Inc. - Inventors: Wilson Harron; Markus K. Cremer - Web page: https://patentgrep.com/patent/US11308731B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 16/783: using metadata automatically derived from the content - G06V 10/56: relating to colour - G06V 20/48: {Matching video sequences} - G06V 40/1365: {Matching; Classification} - G06V 40/50: Maintenance of biometric data or enrolment thereof - G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods of identifying unknown video content are described. An example method may include receiving a first fingerprint and a second fingerprint. The first fingerprint may be a color-based fingerprint derived from colors in a portion of the unknown video content, and the second fingerprint may be at least partially based on a feature other than the colors of the unknown video content. A reference database of reference fingerprints may then be queried using one of the first fingerprint or the second fingerprint to obtain a candidate group of fingerprints. The candidate group of fingerprints may then be queried using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint. The unknown video content may then be identified using the at least one query fingerprint. ## Claims ### Claim 1 A method comprising: determining a first fingerprint for unknown video content, the first fingerprint being a color-based fingerprint based on a set of colors in a frame of the unknown video content; determining a second fingerprint for the unknown video content, the second fingerprint being a patch-based fingerprint based on a plurality of patch-based values, wherein determining the second fingerprint comprises: dividing the frame of the unknown video content into a plurality of regions; selecting a plurality of patches in the frame, wherein selecting the plurality of patches in the frame comprises selecting at least two patches in each region of the plurality of regions; and deriving a patch-based value for each respective patch of the plurality of patches to obtain the plurality of patch-based values; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 2 (depends on 1) The method of claim 1, further comprising receiving the unknown video content at a media playback device including a hardware processor, the query of the at least one reference database and the query of the candidate group of fingerprints being performed by the hardware processor. ### Claim 3 (depends on 2) The method of claim 2, wherein the at least one reference database is stored in memory of the media playback device. ### Claim 4 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database of reference fingerprints comprises sending the first fingerprint and the second fingerprint to a remote server system to execute the query, the at least one reference database being part of the remote server system. ### Claim 5 (depends on 1) The method of claim 1, wherein the at least one reference database comprises: a color-based fingerprint database including reference fingerprints that are color-based; and a patch-based fingerprint database including reference fingerprints that are patch-based. ### Claim 6 (depends on 1) The method of claim 1, wherein the set of colors includes more than 8 colors. ### Claim 7 (depends on 6) The method of claim 6, wherein the set of colors includes 16 colors. ### Claim 8 (depends on 1) The method of claim 1, wherein the at least one reference database is queried with the first fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the second fingerprint to identify the at least one query fingerprint. ### Claim 9 (depends on 1) The method of claim 1, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint. ### Claim 10 (depends on 1) The method of claim 1, wherein the color-based fingerprint is based on a plurality of color-based values, wherein each color-based value is determined for a respective color in the set of colors. ### Claim 11 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database using one of the first fingerprint or the second fingerprint uses a fingerprint defined by a lower number of bits than that used to query the candidate group. ### Claim 12 (depends on 1) The method of claim 1, wherein initiating the query of the at least one reference database comprises querying the at least one reference database of reference fingerprints using the color-based fingerprint. ### Claim 13 (depends on 12) The method of claim 12, wherein querying the at least one reference database using the color-based fingerprint comprises determining that a similarity between the color-based fingerprint and the reference fingerprints satisfies a predetermined threshold associated with vector distance measurement. ### Claim 14 (depends on 1) The method of claim 1, wherein the at least two patches in each region of the plurality of regions include a patch of left-right features in the region and a patch of top-bottom features in the region. ### Claim 15 (depends on 1) The method of claim 1, wherein the plurality of regions is defined by a 4×3 grid, and wherein selecting the plurality of patches in the frame comprises selecting a predetermined number of patches in the frame. ### Claim 16 A computer comprising: one or more hardware processors; and memory including instructions that, when executed by the one or more processors, cause the computer to perform operations comprising: determining a first fingerprint for unknown video content, the first fingerprint being a color-based fingerprint based on a set of colors in a frame of the unknown video content; determining a second fingerprint for the unknown video content, the second fingerprint being a patch-based fingerprint based on a plurality of patch-based values, wherein determining the second fingerprint comprises: dividing the frame of the unknown video content into a plurality of regions; selecting a plurality of patches in the frame, wherein selecting the plurality of patches in the frame comprises selecting at least two patches in each region of the plurality of regions; and deriving a patch-based value for each respective patch of the plurality of patches to obtain the plurality of patch-based values; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 17 (depends on 16) The computer of claim 16, wherein the computer is a media playback device including the at least one reference database, the media playback device configured to receive the unknown video content and query the at least one reference database and the candidate group of fingerprints to identify the at least one query fingerprint. ### Claim 18 (depends on 16) The computer of claim 16, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint. ### Claim 19 A non-transitory computer-readable storage medium embodying instructions which, when executed by a machine, cause the machine to perform operations comprising: determining a first fingerprint for unknown video content, the first fingerprint being a color-based fingerprint based on a set of colors in a frame of the unknown video content; determining a second fingerprint for the unknown video content, the second fingerprint being a patch-based fingerprint based on a plurality of patch-based values, wherein determining the second fingerprint comprises: dividing the frame of the unknown video content into a plurality of regions; selecting a plurality of patches in the frame, wherein selecting the plurality of patches in the frame comprises selecting at least two patches in each region of the plurality of regions; and deriving a patch-based value for each respective patch of the plurality of patches to obtain the plurality of patch-based values; initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the machine is a media playback device, further comprising instructions to receive the unknown video content at the media playback device including a hardware processor, and query the at least one reference database and the candidate group of fingerprints using the hardware processor.