# US 2021/0084345 A1 · Method and System for Re-Uniting Metadata with Media-Stream Content at a Media Client, to Facilitate Action by the Media Client

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0084345 A1
- Document ID: US20210084345A1
- Status: Published application
- Kind: A1
- Application number: 15/929,211
- Filed: 2020-01-28
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: The Nielsen Company (US), LLC
- Applicants: The Nielsen Company (US), LLC
- Inventors: Steven M. Cormie; Brian Jentz
- Web page: https://patentgrep.com/patent/US20210084345A1

## CPC

- H04N 21/235: Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement}
- H04N 21/23892: {involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/83: Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N 21/8358: involving watermark {(protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467)}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8547: involving timestamps for synchronizing content
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In a communication system where a channel feed carrying a media stream is broadcast for delivery of the media stream to a media client, where the channel feed further includes metadata in-band and correlated with a content frame of the media stream, but where the metadata gets removed from the channel feed before the media stream reaches the media client, a computing system (i) extracts the metadata from the channel feed and (ii) transmits out-of-band to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream, the correlation being useable by the media client as a basis to re-correlate the metadata with the frame of the media stream, so as to enable the media client to carry out an action, such as dynamic ad insertion, based on the correlation of the metadata with the frame of the media stream.

## Claims

### Claim 1

In a communication system in which a channel feed carrying a media stream is broadcast for delivery of the media stream to a media client, wherein the channel feed further includes metadata in-band and correlated with a content frame of the media stream, but wherein the metadata gets removed from the channel feed before the media stream reaches the media client, a method comprising:
extracting, by a computing system, the metadata from the channel feed; and
transmitting, by the computing system, in an out-of-band message to the media client, the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream, wherein the correlation of the extracted metadata with the reference fingerprint data representing the frame of the media stream is useable by the media client as a basis to re-correlate the metadata with the frame of the media stream, so as to enable the media client to carry out an action based on the correlation of the metadata with the frame of the media stream.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the computing system determining that the media client does not receive representative metadata with the broadcast channel feed.

### Claim 3 (depends on 2)

The method of claim 2, further comprising the computing system carrying out the extracting and transmitting responsive to the determining that the media client does not receive the representative metadata with the broadcast channel feed.

### Claim 4 (depends on 1)

The method of claim 1, wherein extracting the metadata from the channel feed comprises extracting the metadata from a copy of the channel feed, the copy being provided before the metadata gets removed from the channel feed.

### Claim 5 (depends on 4)

The method of claim 4, wherein the channel feed is provided by a broadcaster and wherein the computing system receives the copy of the channel feed from the broadcaster.

### Claim 6 (depends on 1)

The method of claim 1, wherein the media stream comprises a television channel.

### Claim 7 (depends on 1)

The method of claim 1, wherein the channel feed including the metadata in-band comprises the channel feed including the metadata carried in a transport stream that also carries the media stream, and wherein extracting the metadata from the channel feed comprises reading the metadata from the transport stream.

### Claim 8 (depends on 1)

The method of claim 1, wherein the channel feed including the metadata in-band comprises the channel feed including the metadata in watermarking in the media stream, and wherein extracting the metadata from the channel feed comprises reading the metadata from the watermarking.

### Claim 9 (depends on 1)

The method of claim 1, wherein the channel feed including the metadata in-band comprises the channel feed including the metadata carried in a vertical ancillary data space of a video frame of the media stream, and wherein extracting the metadata from the channel feed comprises reading the metadata from the vertical ancillary data space of the video frame.

### Claim 10 (depends on 1)

The method of claim 1, wherein transmitting to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream comprises transmitting to the media client the extracted metadata timestamped with a reference timestamp and transmitting to the media client the reference fingerprint data timestamped with the same reference timestamp.

### Claim 11 (depends on 1)

The method of claim 1, wherein transmitting to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream comprises transmitting to the media client the extracted metadata attached with the reference fingerprint data.

### Claim 12 (depends on 1)

The method of claim 1,
wherein the action carried out by the media client based on the correlation of the metadata with the frame of the media stream comprises a content-related action, and
wherein the media client uses the correlation of the extracted metadata with the reference fingerprint data representing the frame of the media stream as a basis to re-correlate the metadata with the frame of the media stream, so as to enable the media client to carry out the content-related action based on the correlation of the metadata with the frame of the media stream.

### Claim 13 (depends on 1)

The method of claim 1, wherein the action carried out by the media client based on the correlation of the metadata with the frame of the media stream comprises dynamic ad insertion.

### Claim 14

In a system in which a channel feed carrying a media stream is broadcast for delivery of the media stream to a media client, wherein the channel feed further includes metadata in-band and correlated with a content frame of the media stream, but wherein the metadata gets removed from the channel feed before the media stream reaches the media client, a computing system comprising:
a capture module configured to generate reference fingerprint data representing the frame of the media stream and to extract the metadata from the channel feed; and
one or more modules configured to transmit out-of-band to the media client the generated reference fingerprint data in correlation with the extracted metadata, wherein the correlation of the extracted metadata with the reference fingerprint data representing the frame of the media stream is useable by the media client as a basis to re-correlate the metadata with the frame of the media stream, so as to enable the media client to carry out an action based on the correlation of the metadata with the frame of the media stream.

### Claim 15 (depends on 14)

The computing system of claim 14, wherein the computing system is configured to determine that the media client does not receive representative metadata with the broadcast channel feed, and wherein the computing system is configured to carry out the extracting and transmitting responsive to the determining that the media client does not receive the representative metadata with the broadcast channel feed.

### Claim 16 (depends on 14)

The computing system of claim 14, wherein the channel feed including the metadata in-band comprises the channel feed including the metadata carried in a transport stream that also carries the media stream, and wherein extracting the metadata from the channel feed comprises reading the metadata from the transport stream.

### Claim 17 (depends on 14)

The computing system of claim 14, wherein the channel feed including the metadata in-band comprises the channel feed including the metadata in watermarking in the media stream, and wherein extracting the metadata from the channel feed comprises reading the metadata from the watermarking.

### Claim 18 (depends on 14)

The computing system of claim 14, wherein transmitting to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream comprises transmitting to the media client the extracted metadata timestamped with a reference timestamp and transmitting to the media client the reference fingerprint data timestamped with the same reference timestamp.

### Claim 19 (depends on 14)

The computing system of claim 14, wherein transmitting to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream comprises transmitting to the media client the extracted metadata attached with the reference fingerprint data.

### Claim 20 (depends on 14)

The computing system of claim 14, wherein the action carried out by the media client based on the correlation of the metadata with the frame of the media stream comprises dynamic ad insertion.
