# US 2022/0044694 A1 · Methods and apparatus for rate quality scalable coding with generative models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044694 A1
- Document ID: US20220044694A1
- Status: Published application
- Kind: A1
- Application number: 17/290,193
- Filed: 2019-10-29
- Published: 2022-02-10
- Earliest claimed: 2018-10-29
- Assignees: Dolby International AB
- Applicants: Dolby International AB
- Inventors: Janusz Klejsa; Per Hedelin
- Web page: https://patentgrep.com/patent/US20220044694A1

## CPC

- G10L 19/06: Determination or coding of the spectral characteristics, e.g. of the short-term prediction coefficients
- G10L 19/032: Quantisation or dequantisation of spectral components
- G10L 19/24: Variable rate codecs, e.g. for generating different qualities using a scalable representation such as hierarchical encoding or layered encoding
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Described herein is a method of decoding an audio or speech signal, the method including the steps of: (a) receiving, by a decoder, a coded bitstream including the audio or speech signal and conditioning information; (b) providing, by a bitstream decoder, decoded conditioning information in a format associated with a first bitrate; (c) converting, by a converter, the decoded conditioning information from the format associated with the first bitrate to a format associated with a second bitrate; and (d) providing, by a generative neural network, a reconstruction of the audio or speech signal according to a probabilistic model conditioned by the conditioning information in the format associated with the second bitrate. Described are further an apparatus for decoding an audio or speech signal, a respective encoder, a system of the encoder and the apparatus for decoding an audio or speech signal as well as a respective computer program product.

## Claims

### Claim at position 1

1-29. (canceled)

### Claim 30

A method of decoding an audio or speech signal, the method including the steps of:
(a) receiving, by a receiver, a coded bitstream including the audio or speech signal and conditioning information;
(b) providing, by a bitstream decoder, decoded conditioning information in a format associated with a first bitrate;
(c) converting, by a converter, the decoded conditioning information from the format associated with the first bitrate to a format associated with a second bitrate, the first bitrate being lower than the second bitrate; and
(d) providing, by a generative neural network, a reconstruction of the audio or speech signal according to a probabilistic model conditioned by the decoded conditioning information in the format associated with the second bitrate, wherein the generative neural network reconstructs the signal by performing sampling from a conditional probability density function, which is conditioned using the conditioning information in the format associated with the second bitrate, and wherein the generative neural network is a SampleRNN neural network;
wherein the conditioning information includes an embedded part and a non-embedded part;
wherein the conditioning information includes one or more conditioning parameters;
and wherein a dimensionality, which is defined as a number of the conditioning parameters, of the embedded part of the conditioning information associated with the first bitrate is lower than or equal to the dimensionality of the embedded part of the conditioning information associated with the second bitrate, and wherein the dimensionality of the non-embedded part of the conditioning information associated with the first bitrate is the same as the dimensionality of the non-embedded part of the conditioning information associated with the second bitrate.

### Claim 31 (depends on 30)

The method according to claim 30, wherein the first bitrate is a target bitrate and the second bitrate is a default bitrate.

### Claim 32 (depends on 30)

The method according to claim 30, wherein the one or more conditioning parameters are vocoder parameters.

### Claim 33 (depends on 30)

The method according to claim 30, wherein the one or more conditioning parameters are uniquely assigned to the embedded part and the non-embedded part.

### Claim 34 (depends on 33)

The method according to claim 33, wherein the conditioning parameters of the embedded part include one or more of reflection coefficients from a linear prediction filter, or a vector of subband energies ordered from low frequencies to high frequencies, or coefficients of the Karhunen-Loeve transform, or coefficients of a frequency transform.

### Claim 35 (depends on 33)

The method according to claim 33, wherein step (c) further includes:
(i) extending the dimensionality of the embedded part of the conditioning information associated with the first bitrate to the dimensionality of the embedded part of the conditioning information associated with the second bitrate by means of zero padding; or
(ii) extending the dimensionality of the embedded part of the conditioning information associated with the first bitrate to the dimensionality of the embedded part of the conditioning information associated with the second bitrate by means of predicting any missing conditioning parameters based on the available conditioning parameters of the conditioning information associated with the first bitrate.

### Claim 36 (depends on 33)

The method according to claim 33, wherein step (c) further includes converting, by the converter, the non-embedded part of the conditioning information by copying values of the conditioning parameters from the conditioning information associated with the first bitrate into respective conditioning parameters of the conditioning information associated with the second bitrate.

### Claim 37 (depends on 36)

The method according to claim 36, wherein the conditioning parameters of the non-embedded part of the conditioning information associated with the first bitrate are quantized using a coarser quantizer than for the respective conditioning parameters of the non-embedded part of the conditioning information associated with the second bitrate.

### Claim 38 (depends on 30)

The method according to claim 30, wherein the generative neural network is trained based on conditioning information in the format associated with the second bitrate.

### Claim 39 (depends on 30)

The method according to claim 30, wherein the SampleRNN neural network is a four-tier SampleRNN neural network.

### Claim 40

An apparatus for decoding an audio or speech signal, wherein the apparatus includes:
(a) a receiver for receiving a coded bitstream including the audio or speech signal and conditioning information;
(b) a bitstream decoder for decoding the coded bitstream to obtain decoded conditioning information in a format associated with a first bitrate;
(c) a converter for converting the decoded conditioning information from a format associated with the first bitrate to a format associated with a second bitrate, the first bitrate being lower than the second bitrate; and
(d) a generative neural network for providing a reconstruction of the audio or speech signal according to a probabilistic model conditioned by the decoded conditioning information in the format associated with the second bitrate, wherein the generative neural network reconstructs the signal by performing sampling from a conditional probability density function, which is conditioned using the conditioning information in the format associated with the second bitrate, and wherein the generative neural network is a SampleRNN neural network;
wherein the conditioning information includes an embedded part and a non-embedded part;
wherein the conditioning information includes one or more conditioning parameters;
and wherein a dimensionality, which is defined as a number of the conditioning parameters, of the embedded part of the conditioning information associated with the first bitrate is lower than or equal to the dimensionality of the embedded part of the conditioning information associated with the second bitrate, and wherein the dimensionality of the non-embedded part of the conditioning information associated with the first bitrate is the same as the dimensionality of the non-embedded part of the conditioning information associated with the second bitrate.

### Claim 41 (depends on 40)

The apparatus according to claim 40, wherein the first bitrate is a target bitrate and the second bitrate is a default bitrate.

### Claim 42 (depends on 40)

The apparatus according to claim 40, wherein the one or more conditioning parameters are vocoder parameters.

### Claim 43 (depends on 40)

The apparatus according to claim 40, wherein the one or more conditioning parameters are uniquely assigned to the embedded part and the non-embedded part.

### Claim 44 (depends on 43)

The apparatus according to claim 43, wherein the conditioning parameters of the embedded part include one or more of reflection coefficients from a linear prediction filter, or a vector of subband energies ordered from low frequencies to high frequencies, or coefficients of the Karhunen-Loeve transform, or coefficients of a frequency transform.

### Claim 45 (depends on 43)

The apparatus according to claim 43, wherein the converter is further configured to:
(i) extend the dimensionality of the embedded part of the conditioning information associated with the first bitrate to the dimensionality of the embedded part of the conditioning information associated with the second bitrate by means of zero padding; or
(ii) extend the dimensionality of the embedded part of the conditioning information associated with the first bitrate to the dimensionality of the embedded part of the conditioning information associated with the second bitrate by means of predicting any missing conditioning parameters based on the available conditioning parameters of the conditioning information associated with the first bitrate.

### Claim 46 (depends on 43)

The apparatus according to claim 43, wherein the converter is further configured to convert the non-embedded part of the conditioning information by copying values of the conditioning parameters from the conditioning information associated with the first bitrate into respective conditioning parameters of the conditioning information associated with the second bitrate.

### Claim 47 (depends on 46)

The apparatus according to claim 46, wherein the conditioning parameters of the non-embedded part of the conditioning information associated with the first bitrate are quantized using a coarser quantizer than for the respective conditioning parameters of the non-embedded part of the conditioning information associated with the second bitrate.

### Claim 48 (depends on 40)

The apparatus according to claim 40, wherein the generative neural network is trained based on conditioning information in the format associated with the second bitrate.

### Claim 49 (depends on 40)

The apparatus according to claim 40, wherein the SampleRNN neural network is a four-tier SampleRNN neural network.

### Claim 50

An encoder including a signal analyzer and a bitstream encoder, wherein the encoder is configured to provide at least two operating bitrates, including a first bitrate and a second bitrate, wherein the first bitrate is associated with a lower level of quality of reconstruction than the second bitrate, and wherein the first bitrate is lower than the second bitrate;
wherein the encoder is further configured to provide conditioning information for conditioning of a SampleRNN neural network, the conditioning information being associated with the first bitrate including one or more conditioning parameters uniquely assigned to an embedded part and a non-embedded part of the conditioning information;
and wherein a dimensionality, which is defined as a number of the conditioning parameters, of the embedded part of the conditioning information and of the non-embedded part of the conditioning information is based on the first bitrate.

### Claim 51 (depends on 50)

The encoder according to claim 50, wherein the conditioning parameters of the embedded part include one or more of reflection coefficients from a linear prediction filter, or a vector of subband energies ordered from low frequencies to high frequencies, or coefficients of the Karhunen-Loeve transform, or coefficients of a frequency transform.

### Claim 52 (depends on 50)

The encoder according to claim 50, wherein the first bitrate belongs to a set of multiple operating bitrates.
