# US 11,908,483 B2 · Inter-channel feature extraction method, audio separation method and apparatus, and computing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,483 B2
- Document ID: US11908483B2
- Status: Granted
- Kind: B2
- Application number: 17/401,125
- Filed: 2021-08-12
- Published: 2021-12-02
- Granted: 2024-02-20
- Earliest claimed: 2019-07-24
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Rongzhi Gu; Shixiong Zhang; Lianwu Chen; Yong Xu; Meng Yu; Dan Su; Dong Yu
- Web page: https://patentgrep.com/patent/US11908483B2

## CPC

- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- G10L 21/0308: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 25/03: characterised by the type of extracted parameters
- G10L 25/30: using neural networks
- H04S 3/02: of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
- H04S 5/00: Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04S: STEREOPHONIC SYSTEMS

## Abstract

This application relates to a method of extracting an inter channel feature from a multi-channel multi-sound source mixed audio signal performed at a computing device. The method includes: transforming one channel component of a multi-channel multi-sound source mixed audio signal into a single-channel multi-sound source mixed audio representation in a feature space; performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal to extract inter-channel features; performing a feature fusion on the single-channel multi-sound source mixed audio representation and the inter-channel features; estimating respective weights of sound sources in the single-channel multi-sound source mixed audio representation based on a fused multi-channel multi-sound source mixed audio feature; obtaining respective representations of the plurality of sound sources according to the single-channel multi-sound source mixed audio representation and the respective weights; and transforming the respective representations of the sound sources into respective audio signals of the plurality of sound sources.

## Claims

### Claim 1

An audio separation method of a multi-channel multi-sound source mixed audio signal, performed by a computing device by using an artificial neural network, comprising:
transforming one of a plurality of channel components of the multi-channel multi-sound source mixed audio signal into a single-channel multi-sound source mixed audio representation in a feature space;
performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal to extract a plurality of inter-channel features based on a plurality of different values of an inter-channel dilation coefficient z and/or a plurality of different values of an inter-channel stride p and at least one parallel two-dimensional convolutional layer;
performing a feature fusion on the single-channel multi-sound source mixed audio representation and the plurality of inter-channel features to obtain a fused multi-channel multi-sound source mixed audio feature;
estimating respective weights of a plurality of sound sources in the single-channel multi-sound source mixed audio representation based on the fused multi-channel multi-sound source mixed audio feature;
obtaining respective representations of the plurality of sound sources according to the single-channel multi-sound source mixed audio representation and the respective weights; and
transforming the respective representations of the plurality of sound sources into respective audio signals of the plurality of sound sources.

### Claim 2 (depends on 1)

The method according to claim 1, wherein
the multi-channel multi-sound source mixed audio signal comprises C channel components, each channel component comprising S sampling points, C and S being integers greater than or equal to 2,
each two-dimensional convolutional layer comprising corresponding n two-dimensional convolution kernels of the same size, a length of each two-dimensional convolution kernel being less than or equal to S,
for each value of the inter-channel dilation coefficient z and/or each value of the inter-channel stride p, each two-dimensional convolutional layer performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal and the corresponding n two-dimensional convolution kernels of the same size to generate corresponding n feature maps as a corresponding inter-channel feature, z and n being integers, 1≤z<C, and n≥1.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the artificial neural network is implemented as an end-to-end neural network, and
for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting n′,l′,0=1, wn′,l′,h′=0,n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1 ,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 4 (depends on 2)

The method according to claim 2, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ, σ2), 0=−wn′,l′,0,wn′,l′,h′=0, n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 5 (depends on 2)

The method according to claim 2, wherein for each two-dimensional convolutional layer, then two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ,σ2), wn′,l′,1=−λwn′,l′,0, wn′,l′,h′=0,n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2, λ being a learnable network parameter,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 6 (depends on 2)

The method according to claim 2, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0N(μ, σ2), wn′,l′,1=−λl′wn′,l′,h′=0n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2, λl′ being a learnable network parameter and varying with l′,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 7 (depends on 2)

The method according to claim 2, wherein
the at least one two-dimensional convolutional layer comprises a plurality of parallel two-dimensional convolutional layers, and
for different two-dimensional convolutional layers in the plurality of two-dimensional convolutional layers, the two-dimensional convolution kernel has different sizes.

### Claim 8 (depends on 1)

The method according to claim 1, wherein
the artificial neural network is implemented as an end-to-end neural network, and
the audio separation method further comprises a training phase of training the end-to-end neural network, the training phase comprising:
establishing, based on original audio data si(t)of a plurality of different sound sources, multi-channel multi-sound source mixed audio data for training, i=1, 2, . . . , I, I being a total quantity of the different sound sources; and
training the end-to-end neural network by using the multi-channel multi-sound source mixed audio data based on a permutation invariance training criterion, audio data ŝθ(i)(t) estimated by the end-to-end neural network and the original audio data si(t) being used to calculate scores of paired combinations of estimated audio-original audio for the different sound sources, θ(i) representing a paired combination, a negative value of a score of a paired combination with the highest score being used as a loss function of the end-to-end neural network.

### Claim 9

A computing device, comprising a processor and a memory, the memory storing instructions, the instructions, when executed on the processor, causing the processor to perform a plurality of operations including:
transforming one of a plurality of channel components of the multi-channel multi-sound source mixed audio signal into a single-channel multi-sound source mixed audio representation in a feature space;
performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal to extract a plurality of inter-channel features based on a plurality of different values of an inter-channel dilation coefficient z and/or a plurality of different values of an inter-channel stride p and at least one parallel two-dimensional convolutional layer;
performing a feature fusion on the single-channel multi-sound source mixed audio representation and the plurality of inter-channel features to obtain a fused multi-channel multi-sound source mixed audio feature;
estimating respective weights of a plurality of sound sources in the single-channel multi-sound source mixed audio representation based on the fused multi-channel multi-sound source mixed audio feature;
obtaining respective representations of the plurality of sound sources according to the single-channel multi-sound source mixed audio representation and the respective weights; and
transforming the respective representations of the plurality of sound sources into respective audio signals of the plurality of sound sources.

### Claim 10 (depends on 9)

The computing device according to claim 9, wherein
the multi-channel multi-sound source mixed audio signal comprises C channel components, each channel component comprising S sampling points, C and S being integers greater than or equal to 2,
each two-dimensional convolutional layer comprising corresponding n two-dimensional convolution kernels of the same size, a length of each two-dimensional convolution kernel being less than or equal to S,
for each value of the inter-channel dilation coefficient z and/or each value of the inter-channel stride p, each two-dimensional convolutional layer performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal and the corresponding n two-dimensional convolution kernels of the same size to generate corresponding n feature maps as a corresponding inter-channel feature, z and n being integers, 1≤z<C, and n≥1.

### Claim 11 (depends on 10)

The computing device according to claim 10, wherein the artificial neural network is implemented as an end-to-end neural network, and
for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0=l, wn′,l′,1=−l, wn′,l′, h′=0, n′=1,2, . . . , n, l′=0, 1, . . . , l−1, h′=2,3, . . . , h−1,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 12 (depends on 10)

The computing device according to claim 10, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ, σ2), 0=−wn′,l′,0, wn′,l′,h′=0, n′=1, 2, . . . , n, l′=0, 1, . . . , l−1,h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 13 (depends on 10)

The computing device according to claim 10, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ, σ2), wn′,l′,1=−λwn′,l′,0, wn′,l′,h′=0, n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2, λ being a learnable network parameter,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 14 (depends on 10)

The computing device according to claim 10, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ, σ2), wn′,l′,1=−λl′wn′,l′,h′=0, n′=1, 2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1, N(μ, σ2) being a Gaussian distribution with a mean being μ and a variance being σ2, λl′, being a learnable network parameter and varying with l′,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, 1 being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 15 (depends on 10)

The computing device according to claim 10, wherein
the at least one two-dimensional convolutional layer comprises a plurality of parallel two-dimensional convolutional layers, and
for different two-dimensional convolutional layers in the plurality of two-dimensional convolutional layers, the two-dimensional convolution kernel has different sizes.

### Claim 16 (depends on 9)

The computing device according to claim 9, wherein
the artificial neural network is implemented as an end-to-end neural network, and
the audio separation method further comprises a training phase of training the end-to-end neural network, the training phase comprising:
establishing, based on original audio data si(t)of a plurality of different sound sources, multi-channel multi-sound source mixed audio data for training, i=1, 2, . . . , I, I being a total quantity of the different sound sources; and
training the end-to-end neural network by using the multi-channel multi-sound source mixed audio data based on a permutation invariance training criterion, audio data ŝθ(i)(t) estimated by the end-to-end neural network and the original audio data si(t) being used to calculate scores of paired combinations of estimated audio-original audio for the different sound sources, θ(i) representing a paired combination, a negative value of a score of a paired combination with the highest score being used as a loss function of the end-to-end neural network.

### Claim 17

A non-transitory computer-readable storage medium, storing instructions, the instructions, when executed on a processor, causing the processor to perform a plurality of operations including:
transforming one of a plurality of channel components of the multi-channel multi-sound source mixed audio signal into a single-channel multi-sound source mixed audio representation in a feature space;
performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal to extract a plurality of inter-channel features based on a plurality of different values of an inter-channel dilation coefficient z and/or a plurality of different values of an inter-channel stride p and at least one parallel two-dimensional convolutional layer;
performing a feature fusion on the single-channel multi-sound source mixed audio representation and the plurality of inter-channel features to obtain a fused multi-channel multi-sound source mixed audio feature;
estimating respective weights of a plurality of sound sources in the single-channel multi-sound source mixed audio representation based on the fused multi-channel multi-sound source mixed audio feature;
obtaining respective representations of the plurality of sound sources according to the single-channel multi-sound source mixed audio representation and the respective weights; and
transforming the respective representations of the plurality of sound sources into respective audio signals of the plurality of sound sources.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein
the multi-channel multi-sound source mixed audio signal comprises C channel components, each channel component comprising S sampling points, C and S being integers greater than or equal to 2,
each two-dimensional convolutional layer comprising corresponding n two-dimensional convolution kernels of the same size, a length of each two-dimensional convolution kernel being less than or equal to S,
for each value of the inter-channel dilation coefficient z and/or each value of the inter-channel stride p, each two-dimensional convolutional layer performing a two-dimensional dilated convolution on the multi-channel multi-sound source mixed audio signal and the corresponding n two-dimensional convolution kernels of the same size to generate corresponding n feature maps as a corresponding inter-channel feature, z and n being integers, 1≤z<C, and n≥1.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the artificial neural network is implemented as an end-to-end neural network, and
for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0=1wn′,l′,1=−1, wn′,l′,h′=0, n′=1,2, . . . , n, l′=0, 1, . . . , l−1, h′=2, 3, . . . , h−1,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein for each two-dimensional convolutional layer, the n two-dimensional convolution kernels of the same size are initialized in a training phase of the end-to-end neural network according to the following solution:
letting wn′,l′,0˜N(μ, σ2), 0=−wn′,l′,0, wn′,l′,h′=0, n′=1, 2, . . . , l−1,h′=2, 3, . . . , h−1, N(μ,σ2) being a Gaussian distribution with a mean being μ and a variance being σ2,
h being a height of each two-dimensional convolution kernel in the two-dimensional convolutional layer, l being a length of each two-dimensional convolution kernel in the two-dimensional convolutional layer, wn′,l′,h′ representing a value of an (n′)th two-dimensional convolution kernel at a position (l′, h′).