# US 2019/0318754 A1 · Methods and Systems for End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318754 A1 - Document ID: US20190318754A1 - Status: Published application - Kind: A1 - Application number: 15/983,256 - Filed: 2018-05-18 - Published: 2019-10-17 - Earliest claimed: 2018-04-16 - Assignees: Mitsubishi Electric Research Laboratories, Inc. - Applicants: Mitsubishi Electric Research Laboratories, Inc. - Inventors: Jonathan Le Roux; John R Hershey; Zhongqiu Wang; Gordon P Wichern - Web page: https://patentgrep.com/patent/US20190318754A1 ## CPC - G10L 19/06: Determination or coding of the spectral characteristics, e.g. of the short-term prediction coefficients - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/048: Activation functions - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G10L 19/02: using spectral analysis, e.g. transform vocoders or subband vocoders - G10L 21/0272: Voice signal separating - G10L 25/30: using neural networks - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for an audio signal processing system for transforming an input audio signal. A processor implements steps of a module by inputting an input audio signal into a spectrogram estimator to extract an audio feature sequence, and process the audio feature sequence to output a set of estimated spectrograms. Processing the set of estimated spectrograms and the audio feature sequence using a spectrogram refinement module, to output a set of refined spectrograms. Wherein the processing of the spectrogram refinement module is based on an iterative reconstruction algorithm. Processing the set of refined spectrograms for the one or more target audio signals using a signal refinement module, to obtain the target audio signal estimates. An output interface to output the optimized target audio signal estimates. Wherein the module is optimized by minimizing an error using an optimizer stored in the memory. ## Claims ### Claim 1 An audio signal processing system for transforming an input audio signal, wherein the input audio signal includes a mixture of one or more target audio signals, the audio signal processing system comprising: a memory including stored executable instructions and a stored module, such that the stored module transforms the input audio signal to obtain target audio signal estimates; an input interface to receive the input audio signal; a processor in communication with the memory and the input interface, wherein the processor implements steps of the stored module by a spectrogram estimator of the stored module to extract an audio feature sequence from the input audio signal, and process the audio feature sequence to output a set of estimated spectrograms, wherein the set of estimated spectrograms includes an estimated spectrogram for each target audio signal; a spectrogram refinement module of the stored module to process the set of estimated spectrograms and the audio feature sequence, to output a set of refined spectrograms, such that the set of refined spectrograms includes a refined spectrogram for each target audio signal, and wherein using the spectrogram refinement module is based on an iterative reconstruction algorithm; a signal refinement module of the stored module to process the set of refined spectrograms for the one or more target audio signals, to obtain target audio signal estimates, such that there is an target audio signal estimate for each target audio signal; and an output interface to output the target audio signal estimates, wherein parameters of the stored module are trained using training data by minimizing an error using an optimizer stored in the memory, wherein the error includes one or more of an error on the set of refined spectrograms, an error including a consistency measurement on the set of refined spectrograms, or an error on the target audio signal estimates. ### Claim 2 (depends on 1) The audio signal processing system of claim 1, wherein the spectrogram estimator uses a deep neural network. ### Claim 3 (depends on 1) The audio signal processing system of claim 1, wherein the spectrogram estimator includes a mask estimation module which outputs a mask estimate value for each target audio signal, and a spectrogram estimate output module which uses the mask estimate value for the one or more target audio signals and the input audio signal, to output the estimated spectrogram for each target audio signal. ### Claim 4 (depends on 3) The audio signal processing system of claim 3, wherein at least one mask estimate value is greater than 1. ### Claim 5 (depends on 1) The audio signal processing system of claim 1, wherein the spectrogram refinement module comprises: defining an iterative procedure acting on the set of estimated spectrograms and the input audio feature sequence; unfolding the iterative procedure into a set of layers, such that there is one layer for each iteration of the iterative procedure, and wherein each layer includes a set of fixed network parameters; forming a neural network using fixed network parameters from the sets of fixed network parameters of layers of previous iterations, as variables to be trained, and untying these variables across the layers of previous iterations, by using the variables as separate variables as each variable is separately applicable to their corresponding layer; training the neural network to obtain a trained neural network; and transforming the set of estimated spectrograms and the audio feature sequence using the trained neural network to obtain the set of refined spectrograms. ### Claim 6 (depends on 1) The audio signal processing system of claim 1, wherein the iterative reconstruction algorithm is an iterative phase reconstruction algorithm. ### Claim 7 (depends on 6) The audio signal processing system of claim 6, wherein the iterative phase reconstruction algorithm is the Multiple Input Spectrogram Inversion (MISI) algorithm. ### Claim 8 (depends on 6) The audio signal processing system of claim 6, wherein the iterative phase reconstruction algorithm is the Griffin-Lim algorithm. ### Claim 9 (depends on 1) The audio signal processing system of claim 1, wherein the error on the target audio signal estimates includes a distance between the target audio signal estimates and reference target audio signals. ### Claim 10 (depends on 1) The audio signal processing system of claim 1, wherein the error on the target audio signal estimates includes a distance between the estimated spectrograms of target audio signal and the refined spectrograms of the target audio signals. ### Claim 11 (depends on 1) The audio signal processing system of claim 1, wherein the spectrogram estimator includes a feature extraction module, such that the feature extraction module extracts the input audio signal from the input audio signal. ### Claim 12 (depends on 1) The audio signal processing system of claim 1, wherein a received audio signal includes one or more of one or more speakers, noise, music, environmental sounds, machine sound. ### Claim 13 A method for transforming input audio signals, comprising the steps of: using a module for transforming an input audio signal of the input audio signals, such that the input audio signal includes a mixture of one or more target audio signals, wherein the module transforms the input audio signal, to obtain target audio signal estimates; using a spectrogram estimator of the model, to extract an audio feature sequence from the input audio signal, and process the audio feature sequence to output a set of estimated spectrograms, wherein the set of estimated spectrograms includes an estimated spectrogram for each target audio signal; using a spectrogram refinement module of the module to process the set of estimated spectrograms and the audio feature sequence, to output a set of refined spectrograms, such that the set of refined spectrograms includes a refined spectrogram for each target audio signal, and wherein using the spectrogram refinement module is based on an iterative reconstruction algorithm; using a signal refinement module of the module to process the set of refined spectrograms for the one or more target audio signals, to obtain target audio signal estimates, such that there is an target audio signal estimate for each target audio signal; and outputting the target audio signal estimates, wherein parameters of the stored module are trained using training data by minimizing an error using an optimizer stored in a memory, wherein the error includes one or more of an error on the set of refined spectrograms, an error including a consistency measurement on the set of refined spectrograms, or an error on the target audio signal estimates, and wherein the steps are performed by a processor in communication with an output device and the memory having stored executable instructions, such that the module is stored in the memory. ### Claim 14 (depends on 13) The method of claim 13, wherein the spectrogram estimator includes a mask estimation module which outputs a mask estimate value for each target audio signal, and a spectrogram estimate output module which uses the mask estimate value for the one or more target audio signals and the input audio signal, to output the estimated spectrogram for each target audio signal, wherein at least one mask estimate value is greater than 1. ### Claim 15 (depends on 13) The method of claim 13, wherein the processing of the spectrogram refinement module comprises: defining an iterative procedure acting on the set of estimated spectrograms and the input audio feature sequence; unfolding the iterative procedure into a set of layers, such that there is one layer for each iteration of the iterative procedure, and wherein each layer includes a set of fixed network parameters; forming a neural network using fixed network parameters from the sets of fixed network parameters of layers of previous iterations, as variables to be trained, and untying these variables across the layers of previous iterations, by using the variables as separate variables as each variable is separately applicable to their corresponding layer; training the neural network to obtain a trained neural network; and transforming the set of estimated spectrograms and the audio feature sequence using the trained neural network to obtain the set of refined spectrograms. ### Claim 16 (depends on 13) The method of claim 13, wherein the iterative reconstruction algorithm is an iterative phase reconstruction algorithm. ### Claim 17 (depends on 13) The method of claim 13, wherein the optimizer includes a gradient-descent based algorithm. ### Claim 18 An audio signal processing system for transforming input audio signals, the audio signal processing system comprising: a memory including stored executable instructions and a stored module, such that the stored module transforms the input audio signal of the input audio signals to obtain target audio signal estimates, wherein input audio signal includes a mixture of one or more target audio signals; an input interface to receive the input audio signal; a processor in communication with the memory and the input interface, wherein the processor implements steps of the stored module by a spectrogram estimator of the stored module to extract an audio feature sequence from the input audio signal, and process the audio feature sequence to output a set of estimated spectrograms, wherein the set of estimated spectrograms includes an estimated spectrogram for each target audio signal; a spectrogram refinement module of the stored module to process the set of estimated spectrograms and the audio feature sequence, to output a set of refined spectrograms, such that the set of refined spectrograms includes a refined spectrogram for each target audio signal, and wherein using the spectrogram refinement module is based on an iterative reconstruction algorithm; a signal refinement module of the stored module to process the set of refined spectrograms for the one or more target audio signals, to obtain target audio signal estimates, such that there is an target audio signal estimate for each target audio signal; and an output interface to output the target audio signal estimates, wherein parameters of the stored module are trained using training data by minimizing an error using an optimizer stored in the memory, wherein the error includes an error on the set of refined spectrograms or an error on the target audio signal estimates or both. ### Claim 19 An audio signal processing system for transforming input audio signals, the audio signal processing comprising: a sound detecting device configured to acquire the input audio signal of the input audio signals from an environment; a signal input interface device configured to receive and transmit the input audio signal, wherein the input audio signal includes a mixture of one or more target audio signals; an audio signal processing device configured to process the input audio signal, wherein the audio signal processing device comprises: a hardware processor configured to connected to a memory, the memory configured to input/output data, wherein the hardware processor executes the steps of a module: accessing the module stored in the memory, such that the stored module transforms the input audio signal to obtain target audio signal estimates; a spectrogram estimator of the stored module, to extract an audio feature sequence from the input audio signal, and process the audio feature sequence to output a set of estimated spectrograms, wherein the set of estimated spectrograms includes an estimated spectrogram for each target audio signal; a spectrogram refinement module of the stored module to process the set of estimated spectrograms and the audio feature sequence, to output a set of refined spectrograms, such that the set of refined spectrograms includes a refined spectrogram for each target audio signal, and wherein using the spectrogram refinement module is based on an iterative reconstruction algorithm; a signal refinement module of the stored module to process the set of refined spectrograms for the one or more target audio signals, to obtain target audio signal estimates, such that there is an target audio signal estimate for each target audio signal; and an output interface to output the target audio signal estimates, wherein parameters of the stored module are trained using training data by minimizing an error using an optimizer stored in the memory, wherein the error includes one or more of an error on the set of refined spectrograms, an error including a consistency measurement on the set of refined spectrograms, or an error on the target audio signal estimates. ### Claim 20 (depends on 1) The audio signal processing system of claim 1, wherein the error further includes an error on the set of estimated spectrograms. ### Claim 21 An audio signal processing system for transforming an input audio signal from a single-channel signal, the audio signal processing system includes an end-to-end approach for single-channel speaker-independent multi-speaker speech separation, the audio signal processing system comprising: a memory including stored executable instructions and a stored module, such that the stored module transforms the input audio signal to obtain target audio signal estimates; a processor in communication with the memory, wherein the processor implements steps of the stored module by a spectrogram estimator of the stored module receives the input audio signal from the single-channel signal, to extract an audio feature sequence, and process the audio feature sequence to output a set of estimated spectrograms, wherein the set of estimated spectrograms includes an estimated spectrogram for each target audio signal; a spectrogram refinement module of the stored module to process the set of estimated spectrograms and the audio feature sequence, to output a set of refined spectrograms, such that the set of refined spectrograms includes a refined spectrogram for each target audio signal, and wherein using the spectrogram refinement module is based on an iterative reconstruction algorithm; a signal refinement module of the stored module to process the set of refined spectrograms for the one or more target audio signals, to obtain target audio signal estimates, such that there is an target audio signal estimate for each target audio signal; and an output interface to output the target audio signal estimates, wherein parameters of the stored module are trained using training data by minimizing an error using an optimizer stored in the memory, wherein the error includes one or more of an error on the set of refined spectrograms, an error including a consistency measurement on the set of refined spectrograms, or an error on the target audio signal estimates.