# US 2015/0149183 A1 · Process and Associated System for Separating a Specified Component and an Audio Background Component from an Audio Mixture Signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0149183 A1
- Document ID: US20150149183A1
- Status: Published application
- Kind: A1
- Application number: 14/555,230
- Filed: 2014-11-26
- Published: 2015-05-28
- Earliest claimed: 2013-11-28
- Assignees: Audionamix
- Applicants: Audionamix
- Inventors: Romain Hennequin
- Web page: https://patentgrep.com/patent/US20150149183A1

## CPC

- G10L 19/26: Pre-filtering or post-filtering
- G10L 21/028: using properties of sound source
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Processes are described herein for transforming an audio mixture signal data structure into a specified component data structure and a background component data structure. In the processes described herein, pitch differences between a guide signal and a dialogue component of an audio mixture signal are accounted for by explicit modeling. Processes described herein can involve obtaining an audio guide signal data structure that corresponds to a dubbing of the specified component, determining parametric spectrogram model data structures for spectrograms of the specified component and the background component, estimating parameters of the parametric spectrogram model data structures to produce data structures representing, a temporary specified signal and a temporary background signal, and filtering the audio mixture signal data structure using the data structures representing the temporary specified signal and the temporary background signal in order to produce data structures representing a specified audio signal and an audio background signal.

## Claims

### Claim 1

A method for transforming an audio mixture signal data structure x(t) representing an audio mixture signal having a specified component and a background component into a data structure corresponding to the specified component and a data structure corresponding to the background component, the method comprising:
obtaining a guide signal data structure g(t) corresponding to a dubbing of the specified component and storing the guide signal data structure g(t) at a computer readable medium;
modeling, by a first modeling module, a spectrogram of a specified signal data structure y(t) as a parametric spectrogram data structure {circumflex over (V)}py having a plurality of frames and including, for each of the plurality of frames, a parameter that models a pitch difference between the guide signal data structure g(t) and the specified component;
modeling, by a second modeling module, a spectrogram of a background signal data structure z(t) as a parametric spectrogram data structure {circumflex over (V)}pz;
estimating, by an estimating module, the parameters of the parametric spectrogram data structure {circumflex over (V)}py to produce a temporary specified signal spectrogram data structure Viy for the specified signal data structure y(t);
estimating, by the estimating module, the parameters of the parametric spectrogram data structure {circumflex over (V)}pz to produce a temporary background signal spectrogram data structure Viz for the background signal data structure z(t);
obtaining, from the audio mixture signal data structure x(t), an audio mixture signal constant Q transform (CQT) data structure Vx and storing the CQT data structure Vx at the computer readable medium; and
filtering, to provide a specified audio signal CQT data structure Vy and a background audio signal CQT data structure Vz, the audio mixture signal CQT Vx using the temporary specified signal spectrogram Viy and the temporary background signal spectrogram Viz;
wherein the specified audio signal CQT data structure Vy is the data structure corresponding to the specified component, and
wherein the background audio signal CQT data structure Vz is the data structure corresponding to the background component.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
applying a time-frequency transform to the audio mixture signal data structure x(t) to produce an audio mixture signal spectrogram data structure Vx;
applying a time-frequency transform to the guide signal data structure g(t) to produce a guide signal spectrogram data structure Vg;
applying an inverse time-frequency transform to the specific audio signal CQT data structure Vy to produce a specified signal data structure y(t);
applying an inverse time-frequency transform to the background audio signal CQT data structure Vz to produce a background signal data structure z(t).

### Claim 3 (depends on 1)

The method of claim 1, wherein the parametric spectrogram data structure {circumflex over (V)}pz is based on a non-negative matrix decomposition.

### Claim 4 (depends on 1)

The method of claim 1, wherein the parametric spectrogram data structure {circumflex over (V)}py includes parameters that model a time shift between the guide signal data structure g(t) and the audio mixture signal data structure x(t).

### Claim 5 (depends on 1)

The method of claim 1, wherein the parametric spectrogram data structure {circumflex over (V)}py includes parameters that model an equalization difference between the guide signal data structure g(t) and the audio mixture signal data structure x(t).

### Claim 6 (depends on 1)

The method of claim 1, wherein both estimating parameters of the parametric spectrogram data structure {circumflex over (V)}py and estimating parameters of the parametric spectrogram data structure {circumflex over (V)}pz are performed according to minimization of a cost function (C).

### Claim 7 (depends on 6)

The method of claim 6, wherein the cost function (C) uses a divergence (d) that is the Itakura Saito divergence.

### Claim 8 (depends on 1)

The method of claim 1, wherein estimating the temporary specified signal spectrogram data structure Viy involves estimating parameters of a model parametric spectrogram data structure Vshiftedg=Σφ↓φVgdiag(Pφ,:);
wherein ↓φVg corresponds to a shift, to an audio guide signal spectrogram data structure Vg, of φ time/frequency points down,
wherein P is a matrix data structure that includes the parameter, for each of the plurality of frames, that accounts for a pitch difference between the audio guide signal data structure g(t) and the specified component of the audio mixture signal data structure x(t);
and wherein diag (Pφ,:) is a diagonal matrix data structure having the components of the φth row of P as a main diagonal.

### Claim 9 (depends on 8)

The method of claim 8, wherein estimating the temporary specified signal spectrogram data structure Viy involves estimating parameters of a model parametric spectrogram data structure Vsyncg=VshiftedgS;
wherein S is a matrix data structure that includes parameters for a correction of a time shift between the guide signal data structure g(t) and the audio mixture signal data structure x(t), and
wherein there exists a positive integer w such that, for all pairs of frames (t1, t2), where |t1−t2|>w, St1t2=0.

### Claim 10 (depends on 9)

The method of claim 9, wherein the parametric spectrogram data structure {circumflex over (V)}py has the form {circumflex over (V)}py=diag(E) (Σφφ↓φVgdiag(Pφ,:))S;
wherein diag(E) is a diagonal matrix data structure that includes parameters for a correction of an equalization difference between the guide signal data structure g(t) and the audio mixture signal data structure x(t), and
wherein estimating the temporary specified signal spectrogram data structure Viy involves estimating E, S, and P.

### Claim 11 (depends on 10)

The method of claim 10, wherein estimating the temporary specified signal spectrogram data structure Viy is iterative,
wherein the update rule
P
φ
,
:
←
P
φ
,
:
⊙
E
T

(
V
g
↓
φ
⊙
(
(
V
⊙
V
^
⊙
-
2
)

S
T
)
)
E
T

(
V
g
↓
φ
⊙
(
V
⊙
V
^
⊙
-
1

S
T
)
)
is used for estimating the values of P,
wherein the update rule
S
←
S
⊙
(
∑
φ


diag

(
E
)

V
g
↓
φ

diag

(
P
φ
,
:
)
)
⊙
V
⊙
V
^
⊙
-
2
(
∑
φ

diag

(
E
)


V
g
↓
φ

diag

(
P
φ
,
:
)
)
is used for estimating the values of S,
wherein the update rule
E
←
E
⊙
(
(
∑
φ


V
g
↓
φ

diag

(
P
φ
,
:
)

S
)
⊙
V
⊙
V
^
⊙
-
2
)

1
T
E
⊙
(
(
∑
φ


V
g
↓
φ

diag

(
P
φ
,
:
)

S
)
⊙
V
^
⊙
-
1
)

1
T
is used for estimating the values of E, and
wherein ⊙ is an operator that corresponds to an element-wise product between matrices (or vectors), (.)⊙(.) is an operator that corresponds to element-wise exponentiation of a matrix by a scalar, (.)T is a matrix transposition, and 1T is a T×1 vector with all coefficients equal to 1.

### Claim 12 (depends on 1)

The method of claim 1, wherein estimating the temporary specified signal spectrogram data structure Viy includes:
performing a first estimation that provides, as output, values of each parameter of the model parametric spectrogram data structure {circumflex over (V)}py, and
performing a tracking step that provides an optimized first estimation value for each parameter of the model parametric spectrogram data structure {circumflex over (V)}py.

### Claim 13 (depends on 12)

The method of claim 12, wherein estimating the temporary specified signal spectrogram data structure Viy further includes performing a second estimation in which values of each parameter of the model parametric spectrogram data structure {circumflex over (V)}py are initialized with the optimized first estimation values for each parameter.

### Claim 14 (depends on 1)

The method of claim 1, wherein filtering the audio mixture signal CQT data structure Vx is performed using Wiener filtering.

### Claim 15

A system for transforming an audio mixture signal data structure x(t) representing an audio mixture signal having a specified component and a background component into a data structure corresponding to the specified component and a data structure corresponding to the background component, the system comprising:
a spectrogram computation module configured to:
apply a time-frequency transform to the audio mixture signal data structure x(t) to produce an audio mixture signal spectrogram data structure Vx, and
apply a time-frequency transform to the audio guide signal data structure g(t) to produce an audio guide signal spectrogram data structure Vg;
a first modeling module configured to model a spectrogram of a specified signal data structure y(t) corresponding to the specified component as a parametric spectrogram data structure {circumflex over (V)}py having a plurality of frames and including, for each of the plurality of frames, a parameter that accounts for a pitch difference between the audio guide signal data structure g(t) and the specified component;
a second modeling module configured to model a spectrogram of a background audio signal data structure z(t) corresponding to the background component as a parametric spectrogram data structure {circumflex over (V)}pz;
an estimation module configured to:
produce a temporary specified signal spectrogram data structure Viy by estimating values for the parameters of the model parametric spectrogram data structure {circumflex over (V)}py, and
produce a temporary background audio signal spectrogram data structure Viz by estimating values for parameters of the model parametric spectrogram data structure {circumflex over (V)}pz; and
a filtering module configured to filter an audio mixture signal CQT data structure Vx using the temporary specified signal spectrogram data structure Viy and the temporary background signal spectrogram data structure Viz to provide a specific audio signal CQT data structure Vy and an audio background signal data structure CQT Vz,
wherein the specified audio signal CQT data structure Vy is the data structure corresponding to the specified component, and
wherein the background audio signal CQT data structure Vz is the data structure corresponding to the background component.

### Claim 16 (depends on 15)

The system of claim 15, further comprising a signal determining module configured to determine a specified signal data structure y(t) from the specified audio signal CQT data structure Vy and to determine a background signal data structure z(t) from the audio background signal data structure CQT Vz.

### Claim 17 (depends on 15)

The system of claim 15, wherein the parametric spectrogram data structure {circumflex over (V)}pz is based on a non-negative matrix decomposition.

### Claim 18 (depends on 15)

The system of claim 15, wherein the parametric spectrogram data structure {circumflex over (V)}py includes parameters that model a time shift between the guide signal data structure g(t) and the audio mixture signal data structure x(t).

### Claim 19 (depends on 15)

The system of claim 15, wherein the parametric spectrogram data structure {circumflex over (V)}py includes parameters that model an equalization difference between the guide signal data structure g(t) and the audio mixture signal data structure x(t).

### Claim 20 (depends on 15)

The system of claim 15, wherein both estimating parameters of the parametric spectrogram data structure {circumflex over (V)}py and estimating parameters of the parametric spectrogram data structure {circumflex over (V)}pz are performed according to minimization of a cost function (C).
