# US 12,015,833 B2 · Use of watermarking to trigger fingerprint-related action > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,833 B2 - Document ID: US12015833B2 - Status: Granted - Kind: B2 - Application number: 17/648,638 - Filed: 2022-01-21 - Published: 2022-05-12 - Granted: 2024-06-18 - Earliest claimed: 2019-02-22 - Assignees: Roku, Inc. - Applicants: Roku, Inc. - Inventors: Shashank C. Merchant; John S. Stavropoulos; Benjamin J. Ropke; Steven M. Cormie - Web page: https://patentgrep.com/patent/US12015833B2 ## CPC - H04N 21/8358: involving watermark {(protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467)} - G06F 21/16: Program or content traceability, e.g. by watermarking - G06F 21/64: Protecting data integrity, e.g. using checksums, certificates or signatures - H04N 21/2353: {specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4415: using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning {(cryptography using biological data H04L9/3231; authentication in networks using biometric H04L63/0861)} - H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Use of watermarking in a media stream as a trigger for carrying out a fingerprint-related action, such as starting to generate fingerprints of the media stream, increasing a rate of generation of fingerprints of the media stream, starting to report fingerprints of the media stream, requesting fingerprints, or engaging in a fingerprint comparison. In an implementation, a media client that is processing the media stream could detect the watermark in the media stream and could responsively carry out or cause to be carried out a fingerprint-related action. Or an entity in a media-distribution path along which the media stream flows to a media client could detect the watermark in the media stream and could responsively carry out or cause to be carried out a fingerprint-related action. Further, the watermark could be inserted into the media stream in the first place based on transport-stream carried metadata, among other possibilities. ## Claims ### Claim 1 A method comprising: extracting, by a media client from an out-of-band signal, a watermark payload including an advertisement identifier of an upcoming advertisement in a media stream being processed by the media client, wherein the watermark payload is based on transport stream metadata associated with the media stream; receiving, by the media client, digital reference fingerprints representing the upcoming advertisement corresponding to the advertisement identifier; transitioning from generating digital query fingerprints representing the media stream at a first non-zero fingerprint-generation rate to a second fingerprint-generation rate that is greater than the first non-zero fingerprint-generation rate; and comparing, by the media client, digital query fingerprints generated at the second fingerprint-generation rate with the digital reference fingerprints to determine when in the media stream the upcoming advertisement begins. ### Claim 2 (depends on 1) The method of claim 1, wherein the digital reference fingerprints also represent the media content preceding the upcoming advertisement. ### Claim 3 (depends on 1) The method of claim 1, wherein the media client comprises a streaming media receiver or a television. ### Claim 4 (depends on 1) The method of claim 1, further comprising: engaging in content modification of a modifiable-content segment of the media stream corresponding to the upcoming advertisement; generating digital query fingerprints of the modifiable-content segment during the content modification; and comparing the digital query fingerprints of the modifiable-content segment with the digital reference fingerprints. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining, based on the comparing, that the digital query fingerprints of the modifiable-content segment do not match the digital reference fingerprints; and based at least on the mismatch, stopping the content modification. ### Claim 6 (depends on 5) The method of claim 5, further comprising reverting to play the modifiable-content segment of the media stream. ### Claim 7 (depends on 1) The method of claim 1, wherein the watermark payload comprises a time offset, and the method further comprises establishing a synchronous lock between a current time according to a clock of the media client and the time offset. ### Claim 8 (depends on 1) The method of claim 1, wherein the watermark payload comprises an absolute time, and the method further comprises establishing a time mapping to translate the absolute time to a current time according to a clock of the media client. ### Claim 9 (depends on 1) The method of claim 1, further comprising: receiving, via Internet Protocol (IP) messaging, a message that correlates a reference code in the watermark payload with content-revision characteristics; and performing the content-revision characteristics on the media stream. ### Claim 10 A media client comprising: a processing unit; and non-transitory data storage coupled to the processing unit, wherein the processing unit is configured to: extract, from an out-of-band signal, a watermark payload including an advertisement identifier of an upcoming advertisement in a media stream being processed by the media client, wherein the watermark payload is based on transport stream metadata associated with the media stream; receive digital reference fingerprints representing the upcoming advertisement corresponding to the advertisement identifier; transition from generating digital query fingerprints representing the media stream at a first non-zero fingerprint-generation rate to a second fingerprint-generation rate that is greater than the first non-zero fingerprint-generation rate; and compare digital query fingerprints generated at the second fingerprint-generation rate with the digital reference fingerprints to determine when in the media stream the upcoming advertisement begins. ### Claim 11 (depends on 10) The media client of claim 10, wherein the digital reference fingerprints also represent the media content preceding the upcoming advertisement. ### Claim 12 (depends on 10) The media client of claim 10, wherein the media client comprises a streaming media receiver or a television. ### Claim 13 (depends on 10) The media client of claim 10, wherein the watermark payload comprises a time offset, the processing unit is further configured to establish a synchronous lock between a current time according to a clock of the media client and the time offset. ### Claim 14 (depends on 10) The media client of claim 10, wherein the watermark payload comprises an absolute time, the processing unit is further configured to establish a time mapping to translate the absolute time to a current time according to a clock of the media client. ### Claim 15 (depends on 10) The media client of claim 10, wherein the processing unit is further configured to: receive, via Internet Protocol (IP) messaging, a message that correlates a reference code in the watermark payload with content-revision characteristics; and perform the content-revision characteristics on the media stream. ### Claim 16 A non-transitory computer-readable medium comprising instructions that when executed by a processing unit of a media client, cause the media client to perform operations, the operations comprising: extracting, from an out-of-band signal, a watermark payload including an advertisement identifier of an upcoming advertisement in a media stream being processed by the media client, wherein the watermark payload is based on transport stream metadata associated with the media stream; receiving digital reference fingerprints representing the upcoming advertisement corresponding to the advertisement identifier; transitioning from generating digital query fingerprints representing the media stream at a first non-zero fingerprint-generation rate to a second fingerprint-generation rate that is greater than the first non-zero fingerprint-generation rate; and comparing digital query fingerprints generated at the second fingerprint-generation rate with the digital reference fingerprints to determine when in the media stream the upcoming advertisement begins. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the digital reference fingerprints also represent the media content preceding the upcoming advertisement. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, further comprising: engaging in content modification of a modifiable-content segment of the media stream corresponding to the upcoming advertisement; generating digital query fingerprints of the modifiable-content segment during the content modification; and comparing the digital query fingerprints of the modifiable-content segment with the digital reference fingerprints. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, further comprising: determining, based on the comparing, that the digital query fingerprints of the content-modifiable segment do not match the digital reference fingerprints; and based at least on the mismatch, stopping the content modification. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, further comprising reverting to play the modifiable-content segment of the media stream.