# US 2021/0082444 A1 · Perceptually-based loss functions for audio encoding and decoding based on machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082444 A1
- Document ID: US20210082444A1
- Status: Published application
- Kind: A1
- Application number: 17/046,284
- Filed: 2019-04-10
- Published: 2021-03-18
- Earliest claimed: 2018-04-11
- Assignees: Dolby Laboratories Licensing Corporation
- Applicants: Dolby Laboratories Licensing Corporation
- Inventors: Roy M. FEJGIN; Grant A. DAVIDSON; Chih-Wei WU; Vivek KUMAR
- Web page: https://patentgrep.com/patent/US20210082444A1

## CPC

- G10L 19/022: Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
- G10L 25/30: using neural networks
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H)
- G10L 19/0204: {using subband decomposition}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Computer-implemented methods for training a neural network, as well as for implementing audio encoders and decoders via trained neural networks, are provided. The neural network may receive an input audio signal, generate an encoded audio signal and decode the encoded audio signal. A loss function generating module may receive the decoded audio signal and a ground truth audio signal, and may generate a loss function value corresponding to the decoded audio signal. Generating the loss function value may involve applying a psychoacoustic model. The neural network may be trained based on the loss function value. The training may involve updating at least one weight of the neural network.

## Claims

### Claim 1

A computer-implemented method for training an autoencoder neural network implemented via a control system comprising one or more processors and one or more non-transitory storage media, comprising:
receiving, by the autoencoder neural network, an input audio signal comprising audio data;
generating, by an encoder portion of the autoencoder neural network and based on the input audio signal, an encoded audio signal;
decoding, by a decoder portion of the autoencoder neural network, the encoded audio signal to produce at least one of a decoded audio signal and decoded transform coefficients;
receiving, by a loss function generating module that is implemented via the control system, said at least one of the decoded audio signal and the decoded transform coefficients and a ground truth audio signal;
generating, by the loss function generating module, a loss function value corresponding to said at least one of the decoded audio signal and the decoded transform coefficients; and
training the autoencoder neural network based on the loss function value, wherein the training involves updating at least one weight of the autoencoder neural network,
wherein generating the loss function value comprises applying a psychoacoustic model, which includes calculation of a noise-to-mask ratio.

### Claim 2 (depends on 1)

The method of claim 1, wherein training the autoencoder neural network comprises backward propagation based on the loss function value.

### Claim 3 (depends on 1)

The method of claim 1, wherein training the autoencoder neural network comprises changing a physical state of at least one non-transitory storage medium location corresponding with at least one weight of the autoencoder neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the encoder portion of the autoencoder neural network comprises an input neuron layer and a plurality of hidden neuron layers, wherein the input neuron layer includes more neurons than a final hidden neuron layer.

### Claim 5 (depends on 1)

The method of claim 1, wherein at least some neurons of the encoder portion of the autoencoder neural network are configured with rectified linear unit (ReLU) activation functions.

### Claim 6 (depends on 1)

The method of claim 1, wherein at least some neurons in hidden layers of the decoder portion of the autoencoder neural network are configured with rectified linear unit (ReLU) activation functions and wherein at least some neurons in an output layer of the decoder portion are configured with sigmoidal activation functions.

### Claim 7 (depends on 1)

The method of claim 1, wherein the psychoacoustic model is based, at least in part, on one or more psychoacoustic masking thresholds.

### Claim 8 (depends on 1)

The method of claim 1, wherein the psychoacoustic model involves on one or more of the following:
modeling an outer ear transfer function;
grouping into critical bands;
frequency-domain masking, including but not limited to level-dependent spreading; and
modeling of a frequency-dependent hearing threshold.

### Claim 9 (depends on 8)

The method of claim 8, wherein generating the loss function value comprises:
transforming the decoded audio signal to a frequency domain to generate a frequency transformed decoded audio signal;
transforming the ground truth audio signal to the frequency domain to generate a frequency transformed ground truth audio signal;
applying an outer ear transfer function to the frequency transformed decoded audio signal to generate a first intermediate signal;
applying the outer ear transfer function to the frequency transformed ground truth audio signal to generate a second intermediate signal;
determining the difference between the first intermediate signal and the second intermediate signal to generate a difference signal;
applying a banding operation to the first intermediate signal to generate a plurality of frequency bands of the first intermediate signal;
applying the banding operation to the second intermediate signal to generate a plurality of frequency bands of the second intermediate signal;
applying a frequency masking operation to the plurality of frequency bands of the second intermediate signal to generate a plurality of masked bands;
determining a noise-to-mask ratio using the plurality of frequency bands of the first intermediate signal and the masked band; and
using the noise-to-mask ratio as the loss function value.

### Claim 10 (depends on 1)

The method of claim 1, wherein the loss function involves calculating an average noise-to-masking ratio and wherein the training involves minimizing the average noise-to-masking ratio.

### Claim 11 (depends on 1)

The method of claim 1, wherein the calculation of the noise-to-mask ratio involves a banding process.

### Claim 12 (depends on 11)

The method of claim 11, wherein the banding process is a Zwicker banding process or a Moore banding process.

### Claim 13 (depends on 1)

Audio encoder, comprising the encoder portion of a neural network that has been trained according to the method of claim 1, wherein the audio encoder is further configured to:
receive a currently-input audio signal;
encode the currently-input audio signal in a compressed audio format; and
output an encoded signal in the compressed audio format.

### Claim 14 (depends on 13)

An audio encoding apparatus, comprising:
an interface system; and
a control system comprising one or more processors and one or more non-transitory storage media operatively coupled to the one or more processors, the control system configured to implement an audio encoder according to claim 13.

### Claim 15 (depends on 1)

An audio decoder, comprising the decoder portion of a neural network that has been trained according to the method of claim 1, wherein the audio decoder is further configured to:
receive a currently-input encoded audio signal in a compressed audio format;
decode the currently-input encoded audio signal in a decompressed audio format; and
output a decoded audio signal in the decompressed audio format.

### Claim 16 (depends on 1)

An audio decoding apparatus, comprising:
an interface system;
a control system comprising one or more processors and one or more non-transitory storage media operatively coupled to the one or more processors, the control system configured to implement an audio decoder, the audio decoder comprising the decoder portion of a neural network that as been trained according to the method of claim 1,
wherein the audio decoder is further configured for:
receiving a currently-input encoded audio signal in a compressed audio format;
decoding the currently-input encoded audio signal in a decompressed audio format; and
outputting a decoded audio signal in the decompressed audio format.

### Claim 17 (depends on 16)

A system including the audio decoding apparatus of claim 16, wherein the system further comprises one or more transducers configured for reproducing the decoded audio signal.

### Claim 18 (depends on 1)

One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method of claim 1.

### Claim 19 (depends on 1)

A computer program product having instructions which, when executed by a computing device or system, cause said computing device or system to perform the method according to claim 1.
