# US 2020/0404347 A1 · Use of Steganographically-Encoded Time Information as Basis to Establish a Time Offset, to Facilitate Taking Content-Related Action > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0404347 A1 - Document ID: US20200404347A1 - Status: Published application - Kind: A1 - Application number: 15/929,220 - Filed: 2020-02-06 - Published: 2020-12-24 - Earliest claimed: 2019-06-24 - Assignees: The Nielsen Company (US), LLC - Applicants: The Nielsen Company (US), LLC - Inventors: Benjamin J. Ropke; John S. Stavropoulos; Shashank C. Merchant - Web page: https://patentgrep.com/patent/US20200404347A1 ## CPC - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - G06T 1/0085: {Time domain based watermarking, e.g. watermarks spread over several images} - H04N 21/234: Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs - H04N 21/23892: {involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/44016: {involving splicing one content stream with another content stream, e.g. for substituting a video clip} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N 21/8358: involving watermark {(protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467)} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - H04N 21/8547: involving timestamps for synchronizing content - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system for using steganographically-encoded time information as a basis to control when a client carries out a content-related action. The client processes for presentation one or more linear media streams, each defining a respective time sequence of frames of media content, and at least one of the streams being steganographically encoded with at least one watermark at a respective time-point within linear media stream, the watermark encoding a timestamp of the respective time-point within the linear media stream according to a server clock. The client extracts the timestamp from the watermark and computes a time offset based on a difference between the extracted timestamp and a current time according to a client clock. And the client uses the computed time offset as a basis to determine when the client should carry out a content-related action in a given one of the one or more linear media streams. ## Claims ### Claim 1 A method comprising: processing by a client one or more linear media streams for presentation, each of the one or more linear media streams defining a respective time sequence of frames of media content, and at least one of the linear media streams being steganographically encoded with at least one watermark at a respective time-point within linear media stream, the watermark encoding a timestamp of the respective time-point within the linear media stream according to a server clock; extracting by the client the timestamp from the watermark, and computing by the client a time offset based on a difference between the extracted timestamp and a current time according to a client clock; and using by the client the computed time offset as a basis to determine when the client should carry out a content-related action in a given one of the one or more linear media streams. ### Claim 2 (depends on 1) The method of claim 1, wherein computing the time offset based on the difference between the extracted timestamp and the current time according to the client clock comprises computing the time offset as the difference. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one linear media stream is steganographically encoded with multiple watermarks each at a respective media-stream time-point and each encoding a respective timestamp of the respective media-stream time-point according to the server clock, the method further comprising: extracting by the client from each of the multiple watermarks the respective timestamp, and computing by the client multiple time offsets including, respectively for each extracted respective timestamp, a respective time offset between the extracted respective timestamp and a respective current time according to the client clock, wherein computing by the client the time offset based on a difference between the extracted timestamp and the current time according to the client clock comprises computing by the client a statistical representation of the multiple computed time offsets. ### Claim 4 (depends on 3) The method of claim 3, wherein the statistical measure comprises an average. ### Claim 5 (depends on 1) The method of claim 1, wherein the content-related action comprises dynamic content modification. ### Claim 6 (depends on 5) The method of claim 5, wherein the content-related action comprises dynamic ad replacement. ### Claim 7 (depends on 5) The method of claim 5, further comprising: receiving by the client from a server a directive for the client to carry out the content-related action in the given linear media stream, the directive specifying an action time point when the client should carry out the content-related action, the action time point being defined according to the server clock, wherein using by the client the computed time offset as a basis to determine when the client should carry out the content-related action in the given linear media stream comprises applying by the client the computed time offset as a basis to translate the specified action time point into an associated client time point according to the client clock. ### Claim 8 (depends on 7) The method of claim 7, wherein the applying the computed time offset as the basis to translate the specified action time point into the associated client time point according to the client clock comprises adding the computed time offset to the specified action time point. ### Claim 9 (depends on 7) The method of claim 7, wherein processing by the client the one or more linear media streams for presentation comprises receiving each of the one or more linear media streams distributed by a content distribution system, wherein the server clock defines time according to the content distribution system. ### Claim 10 (depends on 7) The method of claim 7, further comprising: generating, by the client, query fingerprints representing the given linear media stream; transmitting, from the client to the server, the generated query fingerprints to enable the server to identify the given linear media stream. ### Claim 11 A method comprising: receiving by a client an ongoing media stream that includes watermarks steganographically encoding server-timestamps of portions of the media stream from a time perspective of a sever; and for each of one or more of the steganographically encoded watermarks in the media stream being received by the client, extracting by the client the server-timestamp encoded by the watermark and comparing by the client the extracted server-timestamp with a respective client-time from a time perspective of the client, to compute a time offset useable by the client as a basis to determine when the client should carry out a content-related action. ### Claim 12 (depends on 11) The method of claim 11, wherein the content-related action comprises dynamic content modification. ### Claim 13 A media client comprising: a content input interface; a content presentation interface; a network communication interface; a processing unit; non-transitory data storage; and program instructions stored in the non-transitory data storage and executable by the processing unit to carry out operations including: processing one or more linear media streams for presentation by the media client, each of the one or more linear media streams defining a respective time sequence of frames of media content, and at least one of the linear media streams being steganographically encoded with at least one watermark at a respective time-point within linear media stream, the watermark encoding a timestamp of the respective time-point within the linear media stream according to a server clock, extracting the timestamp from the watermark, and computing a time offset based on a difference between the extracted timestamp and a current time according to a media-client clock, and using the computed time offset as a basis to determine when the media client should carry out a content-related action in a given one of the one or more linear media streams. ### Claim 14 (depends on 13) The media client of claim 13, wherein the media client is a television. ### Claim 15 (depends on 13) The media client of claim 13, wherein computing the time offset based on the difference between the extracted timestamp and the current time according to the media-client clock comprises computing the time offset as the difference. ### Claim 16 (depends on 13) The media client of claim 13, wherein the at least one linear media stream is steganographically encoded with multiple watermarks each at a respective media-stream time-point and each encoding a respective timestamp of the respective media-stream time-point according to the server clock, the operations further including: extracting from each of the multiple watermarks the respective timestamp, and computing multiple time offsets including, respectively for each extracted respective timestamp, a respective time offset between the extracted respective timestamp and a respective current time according to the media-client clock, wherein computing the time offset based on a difference between the extracted timestamp and the current time according to the media-client clock comprises computing by the client a statistical representation of the multiple computed time offsets. ### Claim 17 (depends on 13) The media client of claim 13, wherein the content-related action comprises dynamic content modification. ### Claim 18 (depends on 17) The media client of claim 17, wherein the operations further comprise: receiving from a server a directive for the media client to carry out the content-related action in the given linear media stream, the directive specifying an action time point when the media client should carry out the content-related action, the action time point being defined according to the server clock, wherein using the computed time offset as a basis to determine when the media client should carry out the content-related action in the given linear media stream comprises applying the computed time offset as a basis to translate the specified action time point into an associated client time point according to the media-client clock. ### Claim 19 (depends on 17) The media client of claim 17, wherein processing by the media client the one or more linear media streams for presentation comprises receiving each of the one or more linear media streams distributed by a content distribution system, wherein the server clock defines time according to the content distribution system. ### Claim 20 (depends on 17) The media client of claim 17, wherein the dynamic content modification is responsive to fingerprint-based automatic content recognition.