# US 12,192,537 B2 · Temporal signalling for video coding technology

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,192,537 B2
- Document ID: US12192537B2
- Status: Granted
- Kind: B2
- Application number: 18/475,853
- Filed: 2023-09-27
- Published: 2024-03-21
- Granted: 2025-01-07
- Earliest claimed: 2019-03-20
- Assignees: V-Nova International Limited
- Applicants: V-Nova International Limited
- Inventors: Guido Meardi; Ivan Damnjanovic
- Web page: https://patentgrep.com/patent/US12192537B2

## CPC

- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/93: Run-length coding
- H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh
- H04N 19/124: Quantisation
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/18: the unit being a set of transform coefficients
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/33: in the spatial domain
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/60: using transform coding
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An encoder (300) configured to receive an input video (302) comprising respective frames, each frame being divided into a plurality of tiles and each tile being divided into a plurality of blocks. The encoder is configured to generate a base encoded stream (310) using a base encoder (306), determine (334) a temporal mode for one or more further encoded enhancement streams (328) generated using an enhancement encoder and generate the one or more further encoded enhancement streams (328) according to the determined temporal mode. The temporal mode is either a first temporal mode that does not apply non-zero values from a temporal buffer or a second temporal mode that does apply non-zero values from the temporal buffer (332). Generating the one or more further encoded enhancement streams comprises applying a transform (348) to each of a series of blocks. The temporal mode is determined for one or more of a frame, tile or block.

## Claims

### Claim 1

An encoder configured to encode an input video into a plurality of encoded streams, such that the encoded streams may be combined to reconstruct the input video, the encoder configured to:
receive an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks;
generate a base encoded stream using a base encoder;
determine a temporal mode for one or more further encoded enhancement streams for use in reconstructing the input video together with the base encoded stream, the one or more further encoded enhancement streams being generated using an enhancement encoder, wherein the temporal mode is one of a first temporal mode that does not apply non-zero values from a temporal buffer for generating the one or more further encoded enhancement streams and a second temporal mode that does apply non zero values from the temporal buffer for generating the one or more further encoded enhancement streams; and
generate the one or more further encoded enhancement streams based on data derived from the base encoded stream and the input video according to the determined temporal mode,
wherein generating the one or more further encoded enhancement streams comprises applying a transform to each of a series of blocks of the plurality of blocks, and wherein the temporal mode is determined for one or more of a frame, tile, or block of the input video,
wherein the encoder is configured to:
determine the temporal mode for a second frame of the input video, subsequent to a first frame; and
omit a quantized value of a transformed block of the first frame from the one or more further encoded enhancement streams based on the temporal mode determined for the second frame, wherein the encoder is configured to use the temporal mode determined for the second frame to control a comparison between the quantized value and one or more thresholds to determine whether the quantized value is to be omitted.

### Claim 2 (depends on 1)

The encoder of claim 1, wherein the encoder is configured to determine the temporal mode based on a cost function, wherein the cost function incorporates a cost of sending temporal mode signalling data for the temporal mode.

### Claim 3 (depends on 2)

The encoder of claim 2, wherein the cost function comprises a function of the input video and at least one of the one or more further encoded enhancement streams.

### Claim 4 (depends on 2)

The encoder of claim 2, wherein the encoder is configured to evaluate the cost function for a frame of the input video, a tile of the input video and a block of the input video.

### Claim 5 (depends on 2)

The encoder of claim 2, wherein the cost function is based on a difference between a frame of the input video and corresponding frame of a reconstructed version of the input video.

### Claim 6 (depends on 2)

The encoder of claim 2, wherein the encoder is configured to evaluate the cost function by:
encoding the one or more further encoded enhancement streams using each of the first temporal mode and the second temporal mode; and
comparing one or more metrics determined for each of the first temporal mode and the second temporal mode.

### Claim 7 (depends on 2)

The encoder of claim 2, wherein the cost function is based on a sum of absolute differences computation.

### Claim 8 (depends on 1)

The encoder of claim 1, wherein the encoder is configured to encode a first block of elements from a given frame in the input video into the plurality of encoded streams without using a second block of elements from the given frame in the input video wherein the enhancement encoder is different from the base encoder, wherein the enhancement encoder is configured to receive residual data generated from a comparison of data derived from the input video and data derived from the base encoded stream, wherein the one or more further encoded enhancement streams comprise encoded residual data that is decodable to reconstruct one or more further sets of residual data for application to a decoded version of the base encoded stream.

### Claim 9 (depends on 1)

The encoder of claim 1, wherein determining the temporal mode comprises:
obtaining temporal mode metadata for a set of blocks of the plurality of blocks; and
determining the temporal mode to use for encoding the set of blocks based on the temporal mode metadata, wherein the encoder is configured to generate temporal mode signalling data for the set of blocks based on the temporal mode and the temporal mode metadata.

### Claim 10 (depends on 1)

The encoder of claim 1, wherein applying non-zero values from the temporal buffer in the second temporal mode comprises deriving a set of non-zero temporal coefficients from the temporal buffer and using the set of non-zero temporal coefficients to modify a current set of coefficients for generating the one or more further encoded enhancement streams.

### Claim 11 (depends on 1)

The encoder of claim 1, wherein the encoder is configured to assign a respective value to at least one of: a frame-based temporal parameter for a frame of the input video; a tile-based temporal parameter for a tile of the input video; and a block-based temporal parameter for a block of the input video, wherein the value assigned to the frame-based temporal parameter indicates the temporal mode for the frame of the input video, the value assigned to the tile-based temporal parameter indicates the temporal mode for the tile of the input video, and the value assigned to the block-based temporal parameter indicates the temporal mode for the block of the input video.

### Claim 12 (depends on 11)

The encoder of claim 11, wherein at least one of:
a bit-length of the value of the frame-based temporal parameter is one bit;
a bit-length of the value of the tile-based temporal parameter is one bit; and
a bit-length of the value of the block-based temporal parameter is one bit.

### Claim 13 (depends on 11)

The encoder of claim 11, wherein the encoder is configured to assign the value of the block-based temporal parameter for the block of the input video to further indicate the temporal mode for the tile comprising the block.

### Claim 14 (depends on 1)

The encoder of claim 1, wherein the encoder is configured to determine whether to refresh the temporal buffer for a given frame of the input video based on at least one of: a first proportion of blocks of the given frame for which the first temporal mode is to be used for reconstructing the given frame, and a second proportion of the blocks of the given frame for which the second temporal mode is to be used for reconstructing the given frame, wherein the refreshing of the temporal buffer for the given frame comprises setting values within the temporal buffer to zero.

### Claim 15 (depends on 1)

The encoder of claim 1, wherein the encoder is configured to generate temporal mode signalling data indicating the temporal mode for the one or more further encoded enhancement streams for a decoder, optionally wherein the temporal mode signalling data is compressed.

### Claim 16

A method of encoding an input video into a plurality of encoded streams, such that the encoded streams may be combined to reconstruct the input video, the method comprising:
receiving an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks;
generating a base encoded stream using a base encoder;
determining a temporal mode for one or more further encoded enhancement streams for use in reconstructing the input video together with the base encoded stream, the one or more further encoded enhancement streams being generated using an enhancement encoder, wherein the temporal mode is one of a first temporal mode that does not apply non-zero values from a temporal buffer for generating the one or more further encoded enhancement streams and a second temporal mode that does apply non-zero values from the temporal buffer for generating the one or more further encoded enhancement streams;
generating the one or more further encoded enhancement streams based on data derived from the base encoded stream and the input video according to the determined temporal mode, wherein generating the one or more further encoded enhancement streams comprises applying a transform to each of a series of blocks of the plurality of blocks, and wherein the temporal mode is determined for one or more of a frame, tile, or block of the input video;
determining the temporal mode for a second frame of the input video, subsequent to a first frame; and
omitting a quantized value of a transformed block of the first frame from the one or more further encoded enhancement streams based on the temporal mode determined for the second frame, wherein the encoder is configured to use the temporal mode determined for the second frame to control a comparison between the quantized value and one or more thresholds to determine whether the quantized value is to be omitted.

### Claim 17

A non-transitory computer-readable medium having stored therein computer executable instructions that when executed by a processor of a computing system cause the computing system to encode an input video into a plurality of encoded streams, such that the encoded streams may be combined to reconstruct the input video, the computing system caused to:
receive an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks;
generate a base encoded stream using a base encoder;
determine a temporal mode for one or more further encoded enhancement streams for use in reconstructing the input video together with the base encoded stream, the one or more further encoded enhancement streams being generated using an enhancement encoder, wherein the temporal mode is one of a first temporal mode that does not apply non-zero values from a temporal buffer for generating the one or more further encoded enhancement streams and a second temporal mode that does apply non-zero values from the temporal buffer for generating the one or more further encoded enhancement streams;
generate the one or more further encoded enhancement streams based on data derived from the base encoded stream and the input video according to the determined temporal mode, wherein generating the one or more further encoded enhancement streams comprises applying a transform to each of a series of blocks of the plurality of blocks, and wherein the temporal mode is determined for one or more of a frame, tile, or block of the input video;
determining the temporal mode for a second frame of the input video, subsequent to a first frame; and
omitting a quantized value of a transformed block of the first frame from the one or more further encoded enhancement streams based on the temporal mode determined for the second frame, wherein the encoder is configured to use the temporal mode determined for the second frame to control a comparison between the quantized value and one or more thresholds to determine whether the quantized value is to be omitted.