# US 12,217,742 B2 · High fidelity audio super resolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,742 B2
- Document ID: US12217742B2
- Status: Granted
- Kind: B2
- Application number: 17/534,221
- Filed: 2021-11-23
- Published: 2023-05-25
- Granted: 2025-02-04
- Earliest claimed: 2021-11-23
- Assignees: Adobe Inc.; The Trustees of Princeton University
- Applicants: Adobe Inc.; The Trustees of Princeton University
- Inventors: Zeyu Jin; Jiaqi Su; Adam Finkelstein
- Web page: https://patentgrep.com/patent/US12217742B2

## CPC

- G10L 15/16: using artificial neural networks
- G10L 21/038: using band spreading techniques
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G10L 15/063: {Training}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments are disclosed for generating full-band audio from narrowband audio using a GAN-based audio super resolution model. A method of generating full-band audio may include receiving narrow-band input audio data, upsampling the narrow-band input audio data to generate upsampled audio data, providing the upsampled audio data to an audio super resolution model, the audio super resolution model trained to perform bandwidth expansion from narrow-band to wide-band, and returning wide-band output audio data corresponding to the narrow-band input audio data.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving narrow-band input audio data;
dividing the narrow-band input audio data into a plurality of batches;
upsampling each batch of the narrow-band input audio data to generate upsampled audio batch data;
providing the upsampled audio batch data to an audio super resolution model, wherein the audio super resolution model is a generator neural network trained as part of a generative adversarial network (GAN) to generate wide-band audio data from narrow-band audio data;
generating, by the audio super resolution model, wide-band audio batch data corresponding to each batch of the plurality of batches, the wide-band audio batch data including high frequency data not included in the narrow-band input audio data;
combining the wide-band audio batch data corresponding to each batch of the plurality of batches to form wide-band output audio data; and
returning the wide-band output audio data corresponding to the narrow-band input audio data.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the narrow-band input audio data is 8 or 16 kHz audio data and wherein the wide-band output audio data is at least 44 kHz audio data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein training the audio super resolution model comprises:
obtaining training audio data;
generating, by the audio super resolution model, wide-band generated audio based on the training audio data;
analyzing the wide-band generated audio by a plurality of discriminators;
comparing outputs of the plurality of discriminators to the training audio data using a loss function; and
training the audio super resolution model and the plurality of discriminators based on the comparison.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the plurality of discriminators includes a spectrogram discriminator and a plurality of waveform discriminators, wherein a first waveform discriminator operates on the wide-band generated audio and each remaining waveform discriminator operates on a downsampled version of the wide-band generated audio.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein obtaining training audio data further comprises:
obtaining wide-band training audio data; and
generating the training audio data by downsampling the wide-band training audio data to a narrowband range.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein generating the training audio data by downsampling the wide-band training audio data to a narrowband range, further comprises:
augmenting the training audio data with noise randomly selected from a noise dataset.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the narrow-band input audio data includes audio recordings of speech.

### Claim 8

A non-transitory computer-readable storage medium including instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:
receive narrow-band input audio data;
divide the narrow-band input audio data into a plurality of batches;
upsample each batch of the narrow-band input audio data to generate upsampled audio batch data;
provide the upsampled audio batch data to an audio super resolution model, wherein the audio super resolution model is a generator neural network trained as part of a generative adversarial network (GAN) to generate wide-band audio data from narrow-band audio data;
generate, by the audio super resolution model, wide-band audio batch data corresponding to each batch of the plurality of batches, the wide-band audio batch data including high frequency data not included in the narrow-band input audio data;
combine the wide-band audio batch data corresponding to each batch of the plurality of batches to form wide-band output audio data; and
return the wide-band output audio data corresponding to the narrow-band input audio data.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the narrow-band input audio data is 8 or 16 kHz audio data and wherein the wide-band output audio data is at least 44 kHz audio data.

### Claim 10 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein training the audio super resolution model comprises:
obtaining training audio data;
generating, by the audio super resolution model, wide-band generated audio based on the training audio data;
analyzing the wide-band generated audio by a plurality of discriminators;
comparing outputs of the plurality of discriminators to the training audio data using a loss function; and
training the audio super resolution model and the plurality of discriminators based on the comparison.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the plurality of discriminators includes a spectrogram discriminator and a plurality of waveform discriminators, wherein a first waveform discriminator operates on the wide-band generated audio and each remaining waveform discriminator operates on a downsampled version of the wide-band generated audio.

### Claim 12 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein obtaining training audio data further comprises:
obtaining wide-band training audio data; and
generating the training audio data by downsampling the wide-band training audio data to a narrowband range.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein generating the training audio data by downsampling the wide-band training audio data to a narrowband range, further comprises:
augmenting the training audio data with noise randomly selected from a noise dataset.

### Claim 14 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the narrow-band input audio data includes audio recordings of speech.

### Claim 15

A system comprising:
a processor; and
a memory including instructions stored thereon which, when executed by the processor, cause the system to:
receive narrow-band input audio data;
divide the narrow-band input audio data into a plurality of batches;
upsample each batch of the narrow-band input audio data to generate upsampled audio batch data;
provide the upsampled audio batch data to an audio super resolution model, wherein the audio super resolution model is a generator neural network trained as part of a generative adversarial network (GAN) to generate wide-band audio data from narrow-band audio data;
generate, by the audio super resolution model, wide-band audio batch data corresponding to each batch of the plurality of batches, the wide-band audio batch data including high frequency data not included in the narrow-band input audio data;
combine the wide-band audio batch data corresponding to each batch of the plurality of batches to form wide-band output audio data; and
return the wide-band output audio data corresponding to the narrow-band input audio data.

### Claim 16 (depends on 15)

The system of claim 15, wherein the narrow-band input audio data is 8 or 16 kHz audio data and wherein the wide-band output audio data is at least 44 kHz audio data.

### Claim 17 (depends on 15)

The system of claim 15, wherein to train the audio super resolution model, the system is further to:
obtain training audio data;
generate, by the audio super resolution model, wide-band generated audio based on the training audio data;
analyze the wide-band generated audio by a plurality of discriminators;
compare outputs of the plurality of discriminators to the training audio data using a loss function; and
train the audio super resolution model and the plurality of discriminators based on the comparison.