# US 12,010,327 B2 · Block-based low latency rate control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,327 B2
- Document ID: US12010327B2
- Status: Granted
- Kind: B2
- Application number: 18/296,351
- Filed: 2023-04-05
- Published: 2023-11-09
- Granted: 2024-06-11
- Earliest claimed: 2019-09-27
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Felix C. Fernandes
- Web page: https://patentgrep.com/patent/US12010327B2

## CPC

- H04N 19/126: Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
- H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567)
- H04N 19/124: Quantisation
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/18: the unit being a set of transform coefficients
- H04N 19/63: using sub-band based transform, e.g. wavelets
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Block-based, low latency rate control for an encoding system in which a wavelet transform decomposes pixel blocks into subbands stored as subbands in wavelet blocks (WBs) for encoding. Quantization parameters (QPs) for the subbands in each WB are estimated using a method that minimizes wavelet-inverse distortion given a rate bound. For each subband, a rate curve is generated based on an unquantized DCT histogram and bit count statistics for the subband, and a distortion curve is generated based on the unquantized DCT histogram and a distortion estimate for the subband that is estimated using a masked estimator. Once the rate-distortion curves for the subbands are generated, a bisection search may be used to find a point on each curve where the slope is the same for all the curves. The QPs associated with those equally sloped points are the global minimizing QPs for the wavelet block.

## Claims

### Claim 1

An encoding system, comprising:
one or more processors configured to:
apply a biorthogonal wavelet transform to input blocks to decompose the input blocks into a plurality of subbands;
for each input block:
generate a wavelet block comprising a plurality of subblocks, wherein each subblock includes data for one of the plurality of subbands;
apply an encoding technique to generate an encoded wavelet block, wherein the encoding technique compresses each subblock of the wavelet block to generate a compressed subblock for each subblock;
for each subblock, apply a weight mask for the subband represented in the subblock to the subblock and a reconstruction of the compressed subblock to generate a distortion estimate for the subblock, wherein the weight mask computes intra-subband error energy and thus includes intra-subband correlations in the distortion estimate; and
sum the distortion estimates of the subblocks to generate an estimate of total distortion for the encoded wavelet block, wherein the total distortion estimates distortion after decoding and wavelet synthesis is performed on the encoded wavelet block at a target device.

### Claim 2 (depends on 1)

The encoding system as recited in claim 1, wherein the weight mask is an N×N mask.

### Claim 3 (depends on 2)

The encoding system as recited in claim 2, where N=5.

### Claim 4 (depends on 1)

The encoding system as recited in claim 1, wherein the weight mask is a low complexity mask generated by thresholding an N×N mask.

### Claim 5 (depends on 4)

The encoding system as recited in claim 4, wherein the low-complexity mask comprises at least five non-zero weights including weights at center, above center, below center, left of center, and right of center of the mask.

### Claim 6 (depends on 1)

The encoding system as recited in claim 1, wherein the biorthogonal wavelet transform decomposes the input blocks into seven subbands (LLLL, LLLH, LLHL, LLHH, LH, HL, and HH).

### Claim 7 (depends on 1)

The encoding system as recited in claim 1, wherein the weight masks for each of the subbands are stored in a look-up table or hardcoded as multipliers in an ASIC.

### Claim 8 (depends on 1)

The encoding system as recited in claim 1, wherein the encoding technique is HEVC (high-efficiency video coding).

### Claim 9 (depends on 1)

The encoding system as recited in claim 1, wherein, to apply an encoding technique to generate an encoded wavelet block, the one or more processors are configured to:
determine quantization parameters (QPs) for the subblocks of the wavelet blocks; and
encode each subblock of the wavelet blocks using respective QPs to generate the compressed subblocks.

### Claim 10 (depends on 9)

The encoding system as recited in claim 9, wherein, to determine quantization parameters (QPs) for the subblocks of the wavelet blocks, the one or more processors are configured to:
for each subblock of a wavelet block:
generate a rate curve for the subblock based on a histogram of unquantized discrete cosine transform (DCT) coefficients for the subblock and bit count statistics for the subblock;
generate a distortion curve based on the histogram of unquantized DCT coefficients and the distortion estimate for the subblock; and
generate a rate-distortion curve based on the rate curve and the distortion curve for the subblock;
estimate optimal QPs for the subblocks of the wavelet block based on the rate-distortion curves for the subblocks and a rate bound for the wavelet block.

### Claim 11 (depends on 1)

The encoding system as recited in claim 1, further comprising a wireless interface configured to transmit the compressed subblocks over a wireless connection to a receiving device.

### Claim 12

A method, comprising:
performing, by one or more processors:
applying a biorthogonal wavelet transform to input blocks to decompose the input blocks into a plurality of subbands;
for each input block:
generating a wavelet block comprising a plurality of subblocks, wherein each subblock includes data for one of the plurality of subbands;
applying an encoding technique to generate an encoded wavelet block, wherein the encoding technique compresses each subblock of the wavelet block to generate a compressed subblock for each subblock;
for each subblock, applying a weight mask for the subband represented in the subblock to the subblock and a reconstruction of the compressed subblock to generate a distortion estimate for the subblock, wherein the weight mask computes intra-subband error energy and thus includes intra-subband correlations in the distortion estimate; and
summing the distortion estimates of the subblocks to generate an estimate of total distortion for the encoded wavelet block, wherein the total distortion estimates distortion after decoding and wavelet synthesis is performed on the encoded wavelet block at a target device.

### Claim 13 (depends on 12)

The method as recited in claim 12, wherein the weight mask is an N×N mask.

### Claim 14 (depends on 13)

The method as recited in claim 13, where N=5.

### Claim 15 (depends on 12)

The method as recited in claim 12, wherein the weight mask is a low complexity mask generated by thresholding an N×N mask.

### Claim 16 (depends on 15)

The method as recited in claim 15, wherein the low-complexity mask comprises at least five non-zero weights including weights at center, above center, below center, left of center, and right of center of the mask.

### Claim 17 (depends on 12)

The method as recited in claim 12, wherein the biorthogonal wavelet transform decomposes the input blocks into seven subbands (LLLL, LLLH, LLHL, LLHH, LH, HL, and HH).

### Claim 18 (depends on 12)

The method as recited in claim 12, wherein, applying an encoding technique to generate an encoded wavelet block comprises:
determine quantization parameters (QPs) for the subblocks of the wavelet blocks, wherein, to determine quantization parameters (QPs) for the subblocks of the wavelet blocks, the one or more processors are configured to:
for each subblock of a wavelet block:
generating a rate curve for the subblock based on a histogram of unquantized discrete cosine transform (DCT) coefficients for the subblock and bit count statistics for the subblock;
generating a distortion curve based on the histogram of unquantized DCT coefficients and the distortion estimate for the subblock; and
generating a rate-distortion curve based on the rate curve and the distortion curve for the subblock;
estimating optimal QPs for the subblocks of the wavelet block based on the rate-distortion curves for the subblocks and a rate bound for the wavelet block encoding each subblock of the wavelet blocks using respective QPs to generate the compressed subblocks.

### Claim 19

A system, comprising:
a device comprising one or more processors and a display subsystem; and
a base station comprising one or more processors configured to:
apply a biorthogonal wavelet transform to pixel blocks from a current video frame to decompose the pixel blocks into a plurality of subbands;
for each pixel block:
generate a wavelet block comprising a plurality of subblocks, wherein each subblock includes data for one of the plurality of subbands;
apply an encoding technique to generate an encoded wavelet block, wherein the encoding technique compresses each subblock of the wavelet block to generate a compressed subblock for each subblock;
for each subblock, apply a weight mask for the subband represented in the subblock to the subblock and a reconstruction of the compressed subblock to generate a distortion estimate for the subblock, wherein the weight mask computes intra-subband error energy and thus includes intra-subband correlations in the distortion estimate; and
sum the distortion estimates of the subblocks to generate an estimate of total distortion for the encoded wavelet block, wherein the total distortion estimates distortion after decoding and wavelet synthesis is performed on the encoded wavelet block at the device; and
transmit the compressed wavelet blocks over a wireless connection to the device;
wherein the one or more processors of the device are configured to:
decompress the compressed wavelet blocks received from the base station;
perform wavelet synthesis on the decompressed wavelet blocks to reconstruct the pixel data for the video frame; and
provide the reconstructed pixel data to the display subsystem for display.

### Claim 20 (depends on 19)

The system as recited in claim 19, wherein the device is a head-mounted device.