# US 12,763,084 B2 · Deep learning segmentation of audio using magnitude spectrogram

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,763,084 B2
- Document ID: US12763084B2
- Status: Granted
- Kind: B2
- Application number: 18/766,392
- Filed: 2024-07-08
- Published: 2024-12-12
- Granted: 2026-10-13
- Earliest claimed: 2020-10-02
- Assignees: Audioshake, Inc.
- Applicants: Audioshake, Inc.
- Inventors: Luke Miner
- Web page: https://patentgrep.com/patent/US12763084B2

## CPC

- G10L 19/0216: {using wavelet decomposition}
- G06N 3/08: Learning methods
- G10L 21/0272: Voice signal separating
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, system, and computer readable medium for decomposing an audio signal into different isolated sources. The techniques and mechanisms convert an audio signal into K input spectrogram fragments. The fragments are sent into a deep neural network to isolate for different sources. The isolated fragments are then combined to form full isolated source audio signals.

## Claims

### Claim 1

A method for decomposing an audio signal, the method comprising:
transforming an original audio file into a complex spectrogram;
splitting the complex spectrogram into K small fragments along the time dimension;
sending each fragment in the K small fragments through one or more convolutional deep neural networks, the convolutional deep neural networks including one or more convolutional layers, the one or more convolutional layers including a subpixel upsample convolutional layer;
producing a sequence of K mask fragments, wherein the K mask fragments are used to extract individual components from the original audio file;
multiplying the K mask fragments with the complex spectrogram to create a new complex spectrogram; and
transforming the new complex spectrogram into a new audio file associated with a decomposed audio signal.

### Claim 2 (depends on 1)

The method of claim 1, wherein the K mask fragments are concatenated together in order to form a complete mask which is the same length as the complex spectrogram.

### Claim 3 (depends on 1)

The method of claim 1, wherein the K mask fragments are binary masks.

### Claim 4 (depends on 1)

The method of claim 1, wherein transforming the original audio file into the complex spectrogram involves a short-time fourier transform.

### Claim 5 (depends on 1)

The method of claim 1, wherein transforming the new complex spectrogram into the new audio file involves an inverse short time fourier transform.

### Claim 6 (depends on 1)

The method of claim 1, wherein at least one of the one or more deep neural networks includes a series of downsample layers and a series of upsample layers.

### Claim 7 (depends on 1)

The method of claim 1, wherein at least one of the one or more deep neural networks includes an input scale layer and an output scale layer.

### Claim 8 (depends on 1)

The method of claim 1, wherein at least one of the one or more deep neural networks includes a bridge layer comprising a first convolution 2D layer and a second convolution 2D layer and an attention layer.

### Claim 9 (depends on 1)

The method of claim 1, wherein instead of concatenating the K mask fragments together and multiplying the complete mask with the complex spectrogram, each K mask fragment is multiplied with a corresponding complex spectrogram fragment thereby producing a fragment of the new complex spectrogram.

### Claim 10

A system for decomposing an audio signal, the system comprising:
a processor; and
memory storing instructions to cause the processor to execute a method, the method comprising:
transforming an original audio file into a complex spectrogram;
splitting the complex spectrogram into K small fragments along the time dimension;
sending each fragment in the K small fragments through one or more convolutional deep neural networks, the convolutional deep neural networks including one or more convolutional layers, the one or more convolutional layers including a subpixel upsample convolutional layer;
producing a sequence of K mask fragments, wherein the K mask fragments are used to extract individual components from the original audio file;
multiplying the K mask fragments with the complex spectrogram to create a new complex spectrogram; and
transforming the new complex spectrogram into a new audio file.

### Claim 11 (depends on 10)

The system of claim 10, wherein the K mask fragments are concatenated together in order to form a complete mask which is the same length as the complex spectrogram.

### Claim 12 (depends on 10)

The system of claim 10, wherein transforming the original audio file into the complex spectrogram involves a short-time fourier transform.

### Claim 13 (depends on 10)

The system of claim 10, wherein transforming the new complex spectrogram into the new audio file involves an inverse short time fourier transform.

### Claim 14 (depends on 10)

The system of claim 10, wherein at least one of the one or more deep neural networks includes a series of downsample layers and a series of upsample layers.

### Claim 15 (depends on 10)

The system of claim 10, wherein at least one of the one or more deep neural networks includes an input scale layer and an output scale layer.

### Claim 16 (depends on 10)

The system of claim 10, wherein at least one of the one or more deep neural networks includes a bridge layer comprising a first convolution 2D layer and a second convolution 2D layer and an attention layer.

### Claim 17 (depends on 10)

The system of claim 10, wherein instead of concatenating the K mask fragments together and multiplying the complete mask with the complex spectrogram, each K mask fragment is multiplied with a corresponding complex spectrogram fragment thereby producing a fragment of the new complex spectrogram.

### Claim 18

A non-transitory computer readable medium storing instructions to cause a processor to execute a method, the method comprising:
transforming an original audio file into a complex spectrogram;
splitting the complex spectrogram into K small fragments along the time dimension;
sending each fragment in the K small fragments through one or more convolutional deep neural networks, the convolutional deep neural networks including one or more convolutional layers, the one or more convolutional layers including a subpixel upsample convolutional layer;
producing a sequence of K mask fragments, wherein the K mask fragments are used to extract individual components from the original audio file;
multiplying the K mask fragments with the complex spectrogram to create a new complex spectrogram; and
transforming the new complex spectrogram into a new audio file.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the K mask fragments are concatenated together in order to form a complete mask which is the same length as the complex spectrogram.

### Claim 20 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein transforming the original audio file into the complex spectrogram involves a short-time fourier transform.