# US 11,910,034 B2 · Network-based assistance for receiver processing of video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,910,034 B2
- Document ID: US11910034B2
- Status: Granted
- Kind: B2
- Application number: 17/312,883
- Filed: 2019-12-17
- Published: 2022-02-03
- Granted: 2024-02-20
- Earliest claimed: 2018-12-21
- Assignees: Koninklijke KPN N.V.; Nederlandse Organisatie voor toegepast-natuurwetenschappelijk onderzoek TNO
- Applicants: Koninklijke KPN N.V.; Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek TNO
- Inventors: Hans Maarten Stokking; Frank Ter Haar; Hendrikus Nathaniël Hindriks
- Web page: https://patentgrep.com/patent/US11910034B2

## CPC

- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/631: {Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols}
- H04N 21/222: Secondary servers, e.g. proxy server, cable television Head-end {(provisioning of proxy services in data packet switching networks H04L67/56)}
- H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N 21/4424: {Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used (error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04)}
- H04N 21/8547: involving timestamps for synchronizing content
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An intermediary system and method may be provided for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network. The processing of the video data by the receiver system may be dependent on an analysis of the video data. The intermediary system may provide processing assist data to the receiver system which comprises an analysis result or a processing instruction derived from the analysis results. Accordingly, the receiver system may process the video data without a need for the receiver system itself to analyze the video data, thereby offloading computational complexity to the intermediary system. Compared to techniques in which most or all of the processing is performed by the intermediary system, an advantage of continuing to process the video data at the receiver system is that the receiver system may already decode the video stream while the video stream is decoded and/or analyzed by the intermediary system, thereby reducing the delay from transmission to display of the video stream.

## Claims

### Claim 1

A processor system configured for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network, wherein the processing of the video data by the receiver system is dependent on an analysis of the video data and comprises compositing an object into the video data, the processor system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream;
via the network interface, forward the video stream to the receiver system;
via the network interface, provide object data to the receiver system, the object data defining at least part of the object;
sequentially decode the video stream to obtain a series of decoded video data parts;
sequentially analyze the decoded video data parts to obtain an analysis result including a characteristic of a composition of the object into the video data, such as a position and/or orientation of the object;
sequentially generate processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis result;
via the network interface, provide the series of processing assist data to the receiver system as a processing assist data stream to enable the receiver system to process the video data as decoded by the receiver system using the analysis result or the processing instruction provided by the processing assist data, wherein the processor system provides the processing assist data for a video data part to the receiver system after the receiver system has started decoding the video data part.

### Claim 2 (depends on 1)

The processor system according to claim 1, wherein the processor is configured to analyze the decoded video data part by at least one of the group of:
a segmentation technique, whereby the analysis result comprises a segmentation of an object in the decoded video data part;
an object tracking technique, whereby the analysis result comprises a position of an object in the decoded video data part; and
a calibration technique, whereby the analysis result comprises a calibration parameter used in the processing of the video data.

### Claim 3 (depends on 1)

The processor system according to claim 1, wherein the processor is configured to include timing information in the processing assist data, the timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated.

### Claim 4 (depends on 3)

The processor system according to claim 3, wherein the timing information comprises at least one of the group of:
a sequence number; and
a content timestamp.

### Claim 5

A processor system configured for processing video data which is received as a video stream via a network, wherein the processing video data comprises compositing an object into the video data, the processor system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from an intermediary system;
via the network interface, receive object data from the intermediary system, the object data defining at least part of the object;
sequentially decode the video stream to obtain a series of decoded video data parts;
via the network interface, receive a processing assist data stream comprising a series of processing assist data comprising an analysis result of an analysis of the individual ones of the decoded video data parts of the video data, the analysis result including a characteristic of a composition of the object into the video data, such as a position and/or orientation of the object, or a processing instruction derived from the analysis result, wherein the processor system has started decoding of a video data part before the processing assist data of the video data part is provided to the processor system from the intermediary system; and
sequentially process the decoded video data parts to obtain processed video data comprising the compositing of the object into the video data using the analysis result or the processing instruction provided by the processing assist data.

### Claim 6 (depends on 5)

The processor system according to claim 5, wherein the processing assist data comprises a segmentation of an object in the part of the video data, and wherein the processor is configured to use the segmentation of the object for processing video data of the object or video data outside of the object.

### Claim 7 (depends on 5)

The processor system according to claim 5, wherein the processing assist data comprises timing information, the timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated, and wherein the processor is configured to identify the part of the video stream or the decoded video data part on the basis of the timing information and to use the analysis result or the processing instruction provided by the processing assist data specifically for the processing of said part.

### Claim 8

A system comprising an intermediary system and a receiver system, wherein:
the intermediary system is configured for assisting the receiver system in processing video data which is streamed as a video stream to the receiver system via a network, wherein the processing of the video data by the receiver system is dependent on an analysis of the video data and comprises compositing an object into the video data, the intermediary system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from a stream source in the network;
via the network interface, forward the video stream to the receiver system;
via the network interface, provide object data to the receiver system, the object data defining at least part of the object;
sequentially decode the video stream to obtain a series of decoded video data parts;
sequentially analyze the decoded video data parts to obtain an analysis result including a characteristic of a composition of the object into the video data, such as a position and/or orientation of the object;
sequentially generate processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis result;
via the network interface, provide the series of processing assist data as a processing assist data stream to the receiver system, wherein the intermediary system provides the processing assist data for a video data part to the receiver system after the receiver system has started decoding the video data part;
the receiver system is configured for processing video data which is received as a video stream via a network, the receiver system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from the intermediary system;
via the network interface, receive the object data from the receiver system;
sequentially decode the video stream to obtain a series of decoded video data parts;
via the network interface, receive the processing assist data stream comprising the series of processing assist data of the individual ones of the decoded video data parts from the intermediary system, and wherein the processing assist data of a decoded video data part is received from the intermediary system at least partially in parallel to decoding said part of the video data;
sequentially process the decoded video data parts to obtain processed video data comprising the compositing of the object into the video data using the analysis result or the processing instruction provided by the processing assist data.

### Claim 9

A computer-implemented method for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network, wherein the processing is dependent on an analysis of the video data and comprises compositing an object into the video data, the method comprising:
via the network, receiving the video stream;
via the network, forwarding the video stream to the receiver system;
via the network, providing object data to the receiver system, the object data defining at least part of the object;
sequentially decoding the video stream to obtain a series of decoded video data parts;
sequentially analyzing the decoded video data parts to obtain an analysis result including a characteristic of a composition of the object into the video data, such as a position and/or orientation of the object;
sequentially generating processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis results;
via the network, providing the series of processing assist data as a processing assist data stream to the receiver system to enable the receiver system to process the video data as decoded by the receiver system using the analysis result or the processing instruction provided by the processing assist data, wherein the providing the processing assist data for a video data part to the receiver system is performed after the receiver system has started decoding the video data part.

### Claim 10 (depends on 9)

A non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 9.

### Claim 11

A computer-implemented method for processing video data which is received as a video stream via a network, wherein the processing video data comprises compositing an object into the video data, the method comprising:
via the network, receiving the video stream from an intermediary system;
via the network, receive object data from the intermediary system, the object data defining at least part of the object;
sequentially decoding the video stream to obtain a series of decoded video data parts;
receiving a processing assist data stream comprising a series of processing assist data from the intermediary system, the processing assist data comprising an analysis result of the analysis of the individual ones of the decoded video data parts of the video data, the analysis result including a characteristic of a composition of the object into the video data, such as a position and/or orientation of the object, or a processing instruction derived from the analysis result, wherein the decoding of a video data part has started before processing assist data of the video data part is provided from the intermediary system; and
sequentially processing the decoded video data parts comprising the compositing of the object into the video data using the analysis result or the processing instruction provided by the processing assist data.

### Claim 12 (depends on 11)

A non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 11.

### Claim 13

A processor system configured for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network, wherein the processing of the video data by the receiver system is dependent on an analysis of the video data, the processor system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream;
via the network interface, forward the video stream to the receiver system;
sequentially decode the video stream to obtain a series of decoded video data parts;
sequentially analyze the decoded video data parts;
sequentially generate processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis result, wherein the processor is configured to include timing information in the processing assist data, the timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated;
via the network interface, provide the series of processing assist data to the receiver system as a processing assist data stream to enable the receiver system to process the video data as decoded by the receiver system using the analysis result or the processing instruction provided by the processing assist data, wherein the processor system provides the processing assist data for a video data part to the receiver system after the receiver system has started decoding the video data part.

### Claim 14 (depends on 13)

The processor system according to claim 13, wherein the processor is configured to analyze the decoded video data part by at least one of the group of:
a segmentation technique, whereby the analysis result comprises a segmentation of an object in the decoded video data part;
an object tracking technique, whereby the analysis result comprises a position of an object in the decoded video data part; and
a calibration technique, whereby the analysis result comprises a calibration parameter used in the processing of the video data.

### Claim 15 (depends on 13)

The processor system according to claim 13, wherein the timing information comprises at least one of the group of:
a sequence number; and
a content timestamp.

### Claim 16

A processor system configured for processing video data which is received as a video stream via a network, the processor system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from an intermediary system;
sequentially decode the video stream to obtain a series of decoded video data parts;
via the network interface, receive a processing assist data stream comprising a series of processing assist data comprising an analysis result of an analysis of the individual ones of the decoded video data parts of the video data, or a processing instruction derived from the analysis result, wherein the processing assist data includes timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated, wherein the processor system has started decoding of a video data part before the processing assist data of the video data part is provided to the processor system from the intermediary system; and
sequentially process the decoded video data parts to obtain processed video data using the analysis result or the processing instruction provided by the processing assist data, wherein the processor is configured to identify the part of the video stream or the decoded video data part on the basis of the timing information and to use the analysis result or the processing instruction provided by the processing assist data specifically for the processing of said part.

### Claim 17 (depends on 16)

The processor system according to claim 16, wherein the processing assist data comprises a segmentation of an object in the part of the video data, and wherein the processor is configured to use the segmentation of the object for processing video data of the object or video data outside of the object.

### Claim 18

A system comprising an intermediary system and a receiver system, wherein:
the intermediary system is configured for assisting the receiver system in processing video data which is streamed as a video stream to the receiver system via a network, and the processing is dependent on an analysis of the video data, the intermediary system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from a stream source in the network;
via the network interface, forward the video stream to the receiver system;
sequentially decode the video stream to obtain a series of decoded video data parts;
sequentially analyze the decoded video data parts to obtain an analysis result;
sequentially generate processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis result, wherein the processor is configured to include timing information in the processing assist data, the timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated;
via the network interface, provide the series of processing assist data as a processing assist data stream to the receiver system, wherein the intermediary system provides the processing assist data for a video data part to the receiver system after the receiver system has started decoding the video data part;
the receiver system is configured for processing video data which is received as a video stream via a network, the receiver system comprising:
a network interface to the network;
a processor configured to:
via the network interface, receive the video stream from the intermediary system;
sequentially decode the video stream to obtain a series of decoded video data parts;
via the network interface, receive the processing assist data stream comprising the series of processing assist data of the individual ones of the decoded video data parts from the intermediary system, wherein the processing assist data includes the timing information, and wherein the processing assist data of a decoded video data part is received from the intermediary system at least partially in parallel to decoding said part of the video data;
sequentially process the decoded video data parts to obtain processed video data using the analysis result or the processing instruction provided by the processing assist data, wherein the processor is configured to identify the part of the video stream or the decoded video data part on the basis of the timing information and to use the analysis result or the processing instruction provided by the processing assist data specifically for the processing of said part.

### Claim 19

A computer-implemented method for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network, wherein the processing is dependent on an analysis of the video data, the method comprising:
via the network, receiving the video stream;
via the network, forwarding the video stream to the receiver system;
sequentially decoding the video stream to obtain a series of decoded video data parts;
sequentially analyzing the decoded video data parts to obtain an analysis result;
sequentially generating processing assist data for the individual ones of the decoded video data parts to obtain a series of processing assist data comprising the analysis result or a processing instruction derived from the analysis results, wherein the processor is configured to include timing information in the processing assist data, the timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated;
via the network, providing the series of processing assist data as a processing assist data stream to the receiver system to enable the receiver system to process the video data as decoded by the receiver system using the analysis result or the processing instruction provided by the processing assist data, wherein the providing the processing assist data for a video data part to the receiver system is performed after the receiver system has started decoding the video data part.

### Claim 20 (depends on 19)

A non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 19.

### Claim 21

A computer-implemented method for processing video data which is received as a video stream via a network, the method comprising:
via the network, receiving the video stream from an intermediary system;
sequentially decoding the video stream to obtain a series of decoded video data parts;
receiving a processing assist data stream comprising a series of processing assist data from the intermediary system, the processing assist data comprising an analysis result of the analysis of the individual ones of the decoded video data parts of the video data, or a processing instruction derived from the analysis result, wherein the processing assist data includes timing information being indicative of the part of the video stream or the decoded video data part from which the processing assist data was generated, wherein the decoding of a video data part has started before processing assist data of the video data part is provided from the intermediary system; and
sequentially processing the decoded video data parts using the analysis result or the processing instruction provided by the processing assist data, wherein the processor is configured to identify the part of the video stream or the decoded video data part on the basis of the timing information and to use the analysis result or the processing instruction provided by the processing assist data specifically for the processing of said part.

### Claim 22 (depends on 21)

A non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 21.