# US 2025/0168412 A1 · Method, apparatus, and medium for visual data processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0168412 A1
- Document ID: US20250168412A1
- Status: Published application
- Kind: A1
- Application number: 19/034,280
- Filed: 2025-01-22
- Published: 2025-05-22
- Earliest claimed: 2022-07-22
- Assignees: Bytedance Inc.
- Applicants: Bytedance Inc.
- Inventors: Zhaobin ZHANG; Semih ESENLIK; Kai ZHANG; Li ZHANG
- Web page: https://patentgrep.com/patent/US20250168412A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 9/002: {using neural networks}
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- H04N 19/98: Adaptive-dynamic-range coding [ADRC]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is proposed. The method comprises: determining, for a conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data at least based on a first probability distribution parameter of the visual data and a gain parameter, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; and performing the conversion based on the residual representation.

## Claims

### Claim 1

A method for visual data processing, comprising:
determining, for a conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data at least based on a first probability distribution parameter of the visual data and at least one gain parameter, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the at least one gain parameter adjusting a value range of the residual representation; and
performing the conversion based on the residual representation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one bitstream comprises a first bitstream and a second bitstream, the first bitstream comprising arithmetic information of the visual data, the second bitstream comprising probability distribution information of the visual data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the conversion comprises decoding the visual data from the first and second bitstreams, the at least one gain parameter comprises a first gain parameter,
wherein determining the residual representation comprises:
determining the first probability distribution parameter of the visual data based on the second bitstream and at least one model; and
obtaining the residual representation of the visual data based on the first bitstream, the first probability distribution parameter and the first gain parameter; and
wherein performing the conversion comprises:
determining the visual data at least based on the residual representation of the visual data.

### Claim 4 (depends on 3)

The method of claim 3, wherein obtaining the residual representation of the visual data comprises:
determining a first intermediate representation of the residual representation based on the first bitstream, the first probability distribution parameter and a first entropy model; and
determining the residual representation by multiplying the first gain parameter to the first intermediate representation,
wherein the first intermediate representation comprises a plurality of components associated with a plurality of channels of the first entropy model, and the first gain parameter comprises a plurality of gain components, a number of the plurality of gain components being a same number as a number of the plurality of channels.

### Claim 5 (depends on 3)

The method of claim 3, wherein determining the visual data comprises:
determining the second probability distribution representation of the visual data based on a first sample of the visual data and a prediction module;
determining a second sample of the visual data based on the second probability distribution representation of the visual data and the residual representation; and
reconstructing the visual data based on the first and second samples and a synthesis model,
wherein determining the second probability distribution representation comprises:
determining context information of the visual data based on the first sample and a context module;
determining hyper information of the visual data at least based on the second bitstream and a second entropy model, wherein the hyper information is further determined based on a hyper coder; and
determining the second probability distribution representation based on the context information, the hyper information and the prediction module.

### Claim 6 (depends on 3)

The method of claim 3, wherein determining the first probability distribution parameter of the visual data comprises:
determining a variance value of the visual data based on the second bitstream and the at least one model, wherein the at least one model comprises a hyper scale coder, the hyper scale coder being not autoregressive; and
determining the first probability distribution parameter based on the variance value and a first predefined value, the first predefined value being added or multiplied to the variance value.

### Claim 7 (depends on 3)

The method of claim 3, wherein obtaining the residual representation comprises:
determining a second intermediate representation of the visual data based on the first bitstream, the first probability distribution parameter, the first gain parameter; and
determining the residual representation based on the second intermediate representation and a second predefined value, the second predefined value being added or multiplied to the second intermediate representation.

### Claim 8 (depends on 3)

The method of claim 3, wherein the residual representation of the visual data comprises a luma residual representation and a chroma residual representation,
wherein the visual data comprises a luma component and a chroma component, and
wherein determining the visual data at least based on the residual representation of the visual data comprises:
determining the luma component at least based on the luma residual representation; and
determining the chroma component at least based on the chroma residual representation,
wherein determining the chroma component comprises: determining the chroma component based on the chroma residual representation and the luma component, wherein a neural network based postprocessing module is applied to the luma component, and the chroma component is determined based on the postprocessed luma component, wherein the neural network based postprocessing module comprises an inter channel correlation information (ICCI) filter.

### Claim 9 (depends on 8)

The method of claim 8, wherein the luma residual representation is determined based on a first luma probability distribution parameter and the first gain parameter, and/or
wherein the first luma probability distribution parameter is determined based on a first hyper scale coder.

### Claim 10 (depends on 8)

The method of claim 8, wherein the luma component is determined based on the luma residual representation, a first context model and a first prediction module,
wherein the luma component is further based on a first hyper coder.

### Claim 11 (depends on 8)

The method of claim 8, wherein the chroma residual representation is determined based on a first chroma probability distribution parameter and the first gain parameter,
wherein the first chroma probability distribution parameter is determined based on a second hyper scale coder.

### Claim 12 (depends on 8)

The method of claim 8, wherein the chroma component is determined based on the chroma residual representation, a second context model and a second prediction module,
wherein the chroma component is further based on a second hyper coder.

### Claim 13 (depends on 2)

The method of claim 2, wherein the conversion comprises encoding the visual data into the first and second bitstreams, the at least one gain parameter comprises a second gain parameter inverse to a first gain parameter associated with a decoding conversion, and
wherein performing the conversion comprises:
determining the first and second bitstreams at least based on the second gain parameter and the residual representation.

### Claim 14 (depends on 1)

The method of claim 1, wherein an inter channel correlation information (ICCI) filter is applied for at least one component of the visual data.

### Claim 15 (depends on 1)

The method of claim 1, wherein the residual representation comprises a quantized residual representation or a de-quantized residual representation.

### Claim 16 (depends on 1)

The method of claim 1, wherein the at least one gain parameter comprises a plurality of gain vectors, a gain vector in the plurality of gain vectors having a plurality of gain components, the visual data comprising a plurality of components associated with a plurality of channels, a number of the plurality of gain components being a same number as a number of the plurality of channels,
wherein a third gain vector is determined based on a convolution of a first gain vector and a second gain vector in the plurality of gain vectors.

### Claim 17 (depends on 1)

The method of claim 1, wherein the conversion includes encoding the visual data into the at least one bitstream, and/or
wherein the conversion includes decoding the visual data from the at least one bitstream.

### Claim 18

An apparatus for visual data processing comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine, for a conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data at least based on a first probability distribution parameter of the visual data and at least one gain parameter, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the at least one gain parameter adjusting a value range of the residual representation; and
perform the conversion based on the residual representation.

### Claim 19

A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine, for a conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data at least based on a first probability distribution parameter of the visual data and at least one gain parameter, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the at least one gain parameter adjusting a value range of the residual representation; and
perform the conversion based on the residual representation.

### Claim 20

A non-transitory computer-readable recording medium storing at least one bitstream of visual data which is generated by a method performed by an apparatus for visual data processing, wherein the method comprises:
determining a residual representation of the visual data at least based on a first probability distribution parameter of the visual data and at least one gain parameter, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the at least one gain parameter adjusting a value range of the residual representation; and
generating the at least one bitstream based on the residual representation.
