# US 2020/0288159 A1 · Combined residual coding in video coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288159 A1
- Document ID: US20200288159A1
- Status: Published application
- Kind: A1
- Application number: 16/810,680
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-08
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Geert Van der Auwera; Bappaditya Ray; Adarsh Krishnan Ramasubramonian; Muhammed Zeyd Coban; Luong Pham Van; Marta Karczewicz
- Web page: https://patentgrep.com/patent/US20200288159A1

## CPC

- H04N 19/45: {performing compensation of the inverse transform mismatch, e.g. Inverse Discrete Cosine Transform [IDCT] mismatch}
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/157: Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/82: involving filtering within a prediction loop
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A video decoder may apply a first inverse residual modification function to first decoded modified chroma residual data to generate first inverse modified chroma residual data. Additionally, the video decoder may apply a second inverse residual modification function to second decoded modified chroma residual data to generate second inverse modified chroma residual data. The first decoded modified chroma residual data is associated with a first chroma component and the second decoded modified chroma residual data is associated with a second chroma component. The video decoder may reconstruct a block of video data based on the first inverse modified chroma residual data and the second inverse modified chroma residual data.

## Claims

### Claim 1

A method of decoding video data, the method comprising:
generating first decoded modified chroma residual data by applying an inverse transform to convert a first set of transform coefficients of a block of the video data from a transform domain to a sample domain;
generating second decoded modified chroma residual data by applying the inverse transform to convert a second set of transform coefficients of the block from the transform domain to the sample domain;
after generating the first decoded modified chroma residual data, applying a first inverse residual modification function to the first decoded modified chroma residual data to generate first inverse modified chroma residual data;
after generating the second decoded modified chroma residual data, applying a second inverse residual modification function to the second decoded modified chroma residual data to generate second inverse modified chroma residual data, wherein the first decoded modified chroma residual data is associated with a first chroma component and the second decoded modified chroma residual data is associated with a second chroma component; and
reconstructing the block of the video data based on the first inverse modified chroma residual data and the second inverse modified chroma residual data.

### Claim 2 (depends on 1)

The method of claim 1, wherein reconstructing the block comprises:
adding the first inverse modified chroma residual data to first predicted chroma data to obtain first reconstructed chroma data of the block; and
adding the second inverse modified chroma residual data to second predicted chroma data to obtain second reconstructed chroma data of the block.

### Claim 3 (depends on 1)

The method of claim 1, wherein the method further comprises:
determining whether to apply the first inverse residual modification function or a third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
determining whether to apply the second inverse residual modification function or a fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
determining whether to apply the first inverse residual modification function or the third inverse residual modification function comprises determining, based on one or more characteristics of the block, whether to apply the first inverse residual modification function or the third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
determining whether to apply the second inverse residual modification function or the fourth inverse residual modification function comprises determining, based on the one or more characteristics of the block, whether to apply the second inverse residual modification function or the fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the characteristics of the block include one or more of: whether a slice or tile that contains the block is an intra type or an inter type, whether current picture referencing is used with the block, whether a dual or shared coding tree is used with the block, dimensions of the block, or an aspect ratio of the block.

### Claim 6 (depends on 3)

The method of claim 3, wherein:
determining whether to apply the first inverse residual modification function or the third inverse residual modification function comprises determining, based on data signaled in a bitstream, whether to apply the first inverse residual modification function or the third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
determining whether to apply the second inverse residual modification function or the fourth inverse residual modification function comprises determining, based on the data signaled in the bitstream, whether to apply the second inverse residual modification function or the fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the first inverse residual modification function is: invF1(x,y)=(resCr(x,y)′+B10)/M10+(resCb(x,y)′+B11)/M11,
the second inverse residual modification function is: invF2(x,y)=(resCr(x,y)′+B20)/M20−(resCb(x,y)′+B21)/M21,
where resCb(x,y)′ is the first decoded modified chroma residual data, resCr(x,y)′ is the second decoded modified chroma residual data, B10, B11, B20, and B21 are rounding terms, and M10, M11, M20 and N21 are normalizing factors.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
applying an inverse CRS process to the first decoded modified chroma residual data before applying the first inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
applying the inverse CRS process to the second decoded modified chroma residual data before applying the second inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
applying an inverse CRS process to the first decoded modified chroma residual data as part of applying the first inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
applying the inverse CRS process to the second decoded modified chroma residual data as part of applying the second inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 10

A method of encoding video data, the method comprising:
applying a first residual modification function to first chroma residual data of a block of the video data to generate first modified chroma residual data;
encoding the first modified chroma residual data, wherein encoding the first modified chroma residual data comprises, after applying the first residual modification function to the first chroma residual data, applying a forward transform to the first modified chroma residual data to convert the first modified chroma residual data from a sample domain to a transform domain;
applying a second residual modification function to second chroma residual data of the block to generate second modified chroma residual data; and
encoding the second modified chroma residual data, wherein:
the first chroma residual data is associated with a first chroma component and the second chroma residual data is associated with a second chroma component, and
encoding the second modified chroma residual data comprises, after applying the second residual modification function to the second chroma residual data, applying the forward transform to the second modified chroma residual data to convert the second modified chroma residual data from the sample domain to the transform domain.

### Claim 11 (depends on 10)

The method of claim 10, wherein the method further comprises:
determining whether to apply the first residual modification function or a third residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
determining whether to apply the second residual modification function or a fourth residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
determining whether to apply the first residual modification function or the third residual modification function comprises determining, based on one or more characteristics of the block, whether to apply the first residual modification function or the third residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
determining whether to apply the second residual modification function or the fourth residual modification function comprises determining, based on the one or more characteristics of the block, whether to apply the second residual modification function or the fourth residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 13 (depends on 12)

The method of claim 12, wherein the characteristics of the block include one or more of: whether a slice or tile that contains the block is an intra type or an inter type, whether current picture referencing is used with the block, whether a dual or shared coding tree is used with the block, dimensions of the block, or an aspect ratio of the block.

### Claim 14 (depends on 11)

The method of claim 11, wherein the method further comprises:
signaling, in a bitstream, data indicating whether the first residual modification function or the third residual modification function was applied to generate the first modified chroma residual data; and
signaling, in the bitstream, data indicating whether the second inverse residual modification function or the fourth inverse residual modification function was applied to generate the second modified chroma residual data.

### Claim 15 (depends on 10)

The method of claim 10, wherein:
the first residual modification function is: F1(x,y)=(resCb(x,y)+A10)/N10−(resCr(x,y)+A11)/N11,
the second residual modification function is: F2(x,y)=(resCb(x,y)+A20)/N20+(resCr(x,y)+A21)/N21,
where resCb(x,y) is the first chroma residual data, resCr(x,y) is the second chroma residual data, A10, A11, A20, and A21 are rounding terms, and N10, N11, N20 and N21 are normalizing factors.

### Claim 16 (depends on 10)

The method of claim 10, further comprising:
applying a CRS process to the first modified chroma residual data after applying the first residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
applying the CRS process to the second modified chroma residual data after applying the second residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
applying a CRS process to the first chroma residual data as part of applying the first residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
applying the CRS process to the second chroma residual data as part of applying the second residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 18

A device for decoding video data, the device comprising:
a memory to store the video data; and
one or more processors implemented in circuitry, the one or more processors configured to:
generate first decoded modified chroma residual data by applying an inverse transform to convert a first set of transform coefficients of a block of the video data from a transform domain to a sample domain;
generate second decoded modified chroma residual data by applying the inverse transform to convert a second set of transform coefficients of the block from the transform domain to the sample domain;
after generating the first decoded modified chroma residual data, apply a first inverse residual modification function to the first decoded modified chroma residual data to generate first inverse modified chroma residual data;
after generating the second decoded modified chroma residual data, apply a second inverse residual modification function to the second decoded modified chroma residual data to generate second inverse modified chroma residual data, wherein the first decoded modified chroma residual data is associated with a first chroma component and the second decoded modified chroma residual data is associated with a second chroma component; and
reconstruct the block of the video data based on the first inverse modified chroma residual data and the second inverse modified chroma residual data.

### Claim 19 (depends on 18)

The device of claim 18, wherein the one or more processors are configured such that, as part of reconstructing the block, the one or more processors:
add the first inverse modified chroma residual data to first predicted chroma data to obtain first reconstructed chroma data of the block; and
add the second inverse modified chroma residual data to second predicted chroma data to obtain second reconstructed chroma data of the block.

### Claim 20 (depends on 18)

The device of claim 18, wherein the one or more processors are further configured to:
determine whether to apply the first inverse residual modification function or a third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
determine whether to apply the second inverse residual modification function or a fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 21 (depends on 20)

The device of claim 20, wherein:
the one or more processors are configured such that, as part of determining whether to apply the first inverse residual modification function or the third inverse residual modification function, the one or more processors determine, based on one or more characteristics of the block, whether to apply the first inverse residual modification function or the third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
the one or more processors are configured such that, as part of determining whether to apply the second inverse residual modification function or the fourth inverse residual modification function, the one or more processors determine, based on the one or more characteristics of the block, whether to apply the second inverse residual modification function or the fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 22 (depends on 21)

The device of claim 21, wherein the characteristics of the block include one or more of: whether a slice or tile that contains the block is an intra type or an inter type, whether current picture referencing is used with the block, whether a dual or shared coding tree is used with the block, dimensions of the block, or an aspect ratio of the block.

### Claim 23 (depends on 20)

The device of claim 20, wherein:
the one or more processors are configured such that, as part of determining whether to apply the first inverse residual modification function or the third inverse residual modification function, the one or more processors determine, based on data signaled in a bitstream, whether to apply the first inverse residual modification function or the third inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
the one or more processors are configured such that, as part of determining whether to apply the second inverse residual modification function or the fourth inverse residual modification function, the one or more processors determine, based on the data signaled in the bitstream, whether to apply the second inverse residual modification function or the fourth inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 24 (depends on 18)

The device of claim 18, wherein:
the first inverse residual modification function is: invF1(x,y)=(resCr(x,y)′+B10)/M10+(resCb(x,y)′+B11)/M11,
the second inverse residual modification function is: invF2(x,y)=(resCr(x,y)′+B20)/M20−(resCb(x,y)′+B21)/M21,
where resCb(x,y)′ is the first decoded modified chroma residual data, resCr(x,y)′ is the second decoded modified chroma residual data, B10, B11, B20, and B21 are rounding terms, and M10, M11, M20 and N21 are normalizing factors.

### Claim 25 (depends on 18)

The device of claim 18, wherein the one or more processors are further configured to:
apply an inverse CRS process to the first decoded modified chroma residual data before applying the first inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
apply the inverse CRS process to the second decoded modified chroma residual data before applying the second inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 26 (depends on 18)

The device of claim 18, wherein the one or more processors are configured to:
apply an inverse CRS process to the first decoded modified chroma residual data as part of applying the first inverse residual modification function to the first decoded modified chroma residual data to generate the first inverse modified chroma residual data; and
apply the inverse CRS process to the second decoded modified chroma residual data as part of applying the second inverse residual modification function to the second decoded modified chroma residual data to generate the second inverse modified chroma residual data.

### Claim 27 (depends on 18)

The device of claim 18, further comprising a display configured to display decoded video data.

### Claim 28 (depends on 18)

The device of claim 18, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

### Claim 29

A device for encoding video data, the device comprising:
a memory to store the video data; and
one or more processors implemented in circuitry, the one or more processors configured to:
apply a first residual modification function to first chroma residual data of a block of the video data to generate first modified chroma residual data;
encode the first modified chroma residual data, wherein the one or more processors are configured such that, as part of encoding the first modified chroma residual data, the one or more processors, after applying the first residual modification function to the first chroma residual data, apply a forward transform to the first modified chroma residual data to convert the first modified chroma residual data from a sample domain to a transform domain;
apply a second residual modification function to second chroma residual data of the block to generate second modified chroma residual data; and
encode the second modified chroma residual data, wherein:
the first chroma residual data is associated with a first chroma component and the second chroma residual data is associated with a second chroma component, and
the one or more processors are configured such that, as part of encoding the second modified chroma residual data, the one or more processors, after applying the second residual modification function to the second chroma residual data, apply the forward transform to the second modified chroma residual data to convert the second modified chroma residual data from the sample domain to the transform domain.

### Claim 30 (depends on 29)

The device of claim 29, wherein the one or more processors are further configured to:
determine whether to apply the first residual modification function or a third residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
determine whether to apply the second residual modification function or a fourth residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 31 (depends on 30)

The device of claim 30, wherein:
the one or more processors are configured such that, as part of determining whether to apply the first residual modification function or the third residual modification function, the one or more processors determine, based on one or more characteristics of the block, whether to apply the first residual modification function or the third residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
the one or more processors are configured such that, as part of determining whether to apply the second residual modification function or the fourth residual modification function, the one or more processors determine, based on the one or more characteristics of the block, whether to apply the second residual modification function or the fourth residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 32 (depends on 31)

The device of claim 31, wherein the characteristics of the block include one or more of: whether a slice or tile that contains the block is an intra type or an inter type, whether current picture referencing is used with the block, whether a dual or shared coding tree is used with the block, dimensions of the block, or an aspect ratio of the block.

### Claim 33 (depends on 30)

The device of claim 30, wherein the one or more processors are further configured to:
signal, in a bitstream, data indicating whether the first residual modification function or the third residual modification function was applied to generate the first modified chroma residual data; and
signal, in the bitstream, data indicating whether the second inverse residual modification function or the fourth inverse residual modification function was applied to generate the second modified chroma residual data.

### Claim 34 (depends on 29)

The device of claim 29, wherein:
the first residual modification function is: F1(x,y)=(resCb(x,y)+A10)/N10−(resCr(x,y)+A11)/N11,
the second residual modification function is: F2(x,y)=(resCb(x,y)+A20)/N20+(resCr(x,y)+A21)/N21,
where resCb(x,y) is the first chroma residual data, resCr(x,y) is the second chroma residual data, A10, A11, A20, and A21 are rounding terms, and N10, N11, N20 and N21 are normalizing factors.

### Claim 35 (depends on 29)

The device of claim 29, wherein the one or more processors are further configured to:
apply a CRS process to the first modified chroma residual data after applying the first residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
apply the CRS process to the second modified chroma residual data after applying the second residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 36 (depends on 29)

The device of claim 29, wherein the one or more processors are further configured to:
apply a CRS process to the first chroma residual data as part of applying the first residual modification function to the first chroma residual data to generate the first modified chroma residual data; and
apply the CRS process to the second chroma residual data as part of applying the second residual modification function to the second chroma residual data to generate the second modified chroma residual data.

### Claim 37 (depends on 29)

The device of claim 29, wherein the device comprises one or more of a camera, a computer, or a mobile device.

### Claim 38

A device for decoding video data, the device comprising:
means for generating first decoded modified chroma residual data by applying an inverse transform to convert a first set of transform coefficients of a block of the video data from a transform domain to a sample domain;
means for generating second decoded modified chroma residual data by applying the inverse transform to convert a second set of transform coefficients of the block from the transform domain to the sample domain;
means for applying, after generating the first decoded modified chroma residual data, a first inverse residual modification function to the first decoded modified chroma residual data to generate first inverse modified chroma residual data;
means for applying, after generating the second decoded modified chroma residual data, a second inverse residual modification function to the second decoded modified chroma residual data to generate second inverse modified chroma residual data, wherein the first decoded modified chroma residual data is associated with a first chroma component and the second decoded modified chroma residual data is associated with a second chroma component; and
means for reconstructing the block of the video data based on the first inverse modified chroma residual data and the second inverse modified chroma residual data.

### Claim 39

A device for encoding video data, the device comprising:
means for applying a first residual modification function to first chroma residual data of a block of the video data to generate first modified chroma residual data;
means for encoding the first modified chroma residual data, wherein encoding the first modified chroma residual data comprises means for applying, after applying the first residual modification function to the first chroma residual data, a forward transform to the first modified chroma residual data to convert the first modified chroma residual data from a sample domain to a transform domain;
means for applying a second residual modification function to second chroma residual data of the block to generate second modified chroma residual data; and
means for encoding the second modified chroma residual data, wherein:
the first chroma residual data is associated with a first chroma component and the second chroma residual data is associated with a second chroma component, and
the means for encoding the second modified chroma residual data comprises means for applying, after applying the second residual modification function to the second chroma residual data, the forward transform to the second modified chroma residual data to convert the second modified chroma residual data from the sample domain to the transform domain.

### Claim 40

A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to:
generate first decoded modified chroma residual data by applying an inverse transform to convert a first set of transform coefficients of a block of video data from a transform domain to a sample domain;
generate second decoded modified chroma residual data by applying the inverse transform to convert a second set of transform coefficients of the block from the transform domain to the sample domain;
after generating the first decoded modified chroma residual data, apply a first inverse residual modification function to the first decoded modified chroma residual data to generate first inverse modified chroma residual data;
after generating the second decoded modified chroma residual data, apply a second inverse residual modification function to the second decoded modified chroma residual data to generate second inverse modified chroma residual data, wherein the first decoded modified chroma residual data is associated with a first chroma component and the second decoded modified chroma residual data is associated with a second chroma component; and
reconstruct the block of the video data based on the first inverse modified chroma residual data and the second inverse modified chroma residual data.

### Claim 41

A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to:
apply a first residual modification function to first chroma residual data of a block of video data to generate first modified chroma residual data;
encode the first modified chroma residual data, wherein as part of causing the one or more processors to encode the first modified chroma residual data, execution of the instructions causes the one or more processors to apply, after applying the first residual modification function to the first chroma residual data, a forward transform to the first modified chroma residual data to convert the first modified chroma residual data from a sample domain to a transform domain;
apply a second residual modification function to second chroma residual data of the block to generate second modified chroma residual data; and
encode the second modified chroma residual data, wherein:
the first chroma residual data is associated with a first chroma component and the second chroma residual data is associated with a second chroma component, and
as part of causing the one or more processors encode the second modified chroma residual data, execution of the instructions causes the one or more processors to apply, after applying the second residual modification function to the second chroma residual data, the forward transform to the second modified chroma residual data to convert the second modified chroma residual data from the sample domain to the transform domain.