# US 2019/0172476 A1 · Deep learning driven multi-channel filtering for speech enhancement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172476 A1 - Document ID: US20190172476A1 - Status: Published application - Kind: A1 - Application number: 15/830,955 - Filed: 2017-12-04 - Published: 2019-06-06 - Earliest claimed: 2017-12-04 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jason Wung; Mehrez Souden; Ramin Pishehvar; Joshua D. Atkins - Web page: https://patentgrep.com/patent/US20190172476A1 ## CPC - G10L 21/0364: for improving intelligibility - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 21/0232: Processing in the frequency domain - G10L 25/03: characterised by the type of extracted parameters - G10L 25/30: using neural networks - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2021/02082: {the noise being echo, reverberation of the speech} - G10L 2021/02166: {Microphone arrays; Beamforming} - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract A number of features are extracted from a current frame of a multi-channel speech pickup and from side information that is a linear echo estimate, a diffuse signal component, or a noise estimate of the multi-channel speech pickup. A DNN-based speech presence probability is produced for the current frame, where the SPP value is produced in response to the extracted features being input to the DNN. The DNN-based SPP value is applied to configure a multi-channel filter whose input is the multi-channel speech pickup and whose output is a single audio signal. In one aspect, the system is designed to run online, at low enough latency for real time applications such voice trigger detection. Other aspects are also described and claimed. ## Claims ### Claim 1 A method for performing speech signal enhancement using a deep neural network, DNN, comprising: extracting a plurality of features from i) a current frame of a multi-channel speech pickup and ii) side information selected from a linear echo estimate, a diffuse signal component, or a noise estimate of the multi-channel speech pickup; producing, as an output of a DNN, a DNN-based speech presence probability, SPP, value for the current frame, wherein the SPP value is produced by the DNN in response to the extracted plurality of features being input to the DNN; and applying the DNN-based SPP value to configure a multi-channel filter whose input is the multi-channel speech pickup and whose output is a single audio signal. ### Claim 2 (depends on 1) The method of claim 1 further comprising steering by the multi-channel filter a beamforming null in a direction of an undesired sound source. ### Claim 3 (depends on 2) The method of claim 2 further comprising steering by the multi-channel filter a unit gain beam or lobe in a direction of a target speech source. ### Claim 4 (depends on 1) The method of claim 1 wherein the multi-channel speech pickup comprises a plurality of acoustic microphone signals produced by a plurality of acoustic microphones, respectively, that are integrated into a housing of a loudspeaker cabinet. ### Claim 5 (depends on 1) The method of claim 1 wherein the plurality of features are extracted from windowed short time Fourier transform domain, or from Mel-frequency bins of a perceptually weighted domain, or from a gammatone filter bank of a perceptually weighted domain, of the multi-channel pickup. ### Claim 6 (depends on 1) The method of claim 1 further comprising performing acoustic echo cancellation upon a plurality of microphone signals, to produce i) the multi-channel speech pickup as a plurality of echo cancelled audio signals and ii) a plurality of linear echo estimates associated with the echo cancelled audio signals. ### Claim 7 (depends on 6) The method of claim 6 wherein the side information comprises the plurality of linear echo estimates, and wherein the multi-channel filter is configured to perform multi-channel residual echo suppression upon the multi-channel speech pickup. ### Claim 8 (depends on 6) The method of claim 6 further comprising performing dereverberation upon the plurality of echo cancelled audio signals, to produce the multi-channel speech pickup. ### Claim 9 (depends on 6) The method of claim 6 wherein the acoustic echo cancellation receives as input i) a first plurality of playback reference signals that represent inputs to a first plurality of speaker drivers in a first loudspeaker cabinet, and ii) a second plurality of playback reference signals that represent inputs to a second plurality of speaker drivers in a second loudspeaker cabinet. ### Claim 10 (depends on 1) The method of claim 1 further comprising applying the SPP value as a spectral mask to the output of the multi-channel filter, prior to transforming the output of the multi-channel filter into time domain. ### Claim 11 (depends on 1) The method of claim 1 further comprising extracting a second plurality of features from i) a new current frame of the multi-channel speech pickup and ii) second side information being a diffuse signal component of the multi-channel speech pickup or a noise estimate for the multi-channel speech pickup, produce, as an output of a second DNN, a second DNN-based speech presence probability value, second SPP value, for the new current frame, wherein the second SPP value is produced by the second DNN in response to the extracted second plurality of features being input to the second DNN, and apply the second SPP value to configure the multi-channel filter. ### Claim 12 A system for performing speech signal enhancement using a deep neural network, DNN, comprising: a processor; and memory having stored therein instructions that when executed by the processor extract a first plurality of features from i) a current frame of a multi-channel speech pickup and ii) first side information selected from the group consisting of a linear echo estimate, a diffuse signal component of the multi-channel speech pickup, or a noise estimate for the multi-channel speech pickup, produce, as an output of a first DNN, a first DNN-based speech presence probability value, first SPP value, for the current frame, wherein the first SPP value is produced by the first DNN in response to the extracted first plurality of features being input to the first DNN, and apply the first SPP value to configure a multi-channel filter whose input is the multi-channel speech pickup and whose output is a single audio signal. ### Claim 13 (depends on 12) The system of claim 12 wherein the memory has stored therein further instructions which when executed by the processor perform acoustic echo cancellation upon a plurality of microphone signals, to produce i) the multi-channel speech pickup as a plurality of echo cancelled audio signals and ii) a plurality of linear echo estimates associated with the echo cancelled audio signals, wherein the first side information comprises the plurality of linear echo estimates, and wherein the multi-channel filter is configured to perform multi-channel residual echo suppression upon the multi-channel speech pickup as the plurality of echo cancelled audio signals. ### Claim 14 (depends on 13) The system of claim 13 wherein the memory has stored therein further instructions which when executed by the processor extract a second plurality of features from i) a new current frame of the multi-channel speech pickup and ii) second side information being a diffuse signal component of the multi-channel speech pickup or a noise estimate for the multi-channel speech pickup, produce, as an output of a second DNN, a second DNN-based speech presence probability value, second SPP value, for the new current frame, wherein the second SPP value is produced by the second DNN in response to the extracted second plurality of features being input to the second DNN, and apply the second SPP value to configure the multi-channel filter. ### Claim 15 (depends on 14) The system of claim 14 wherein i) in response to detecting an absence of playback reference signals the second DNN is selected to produce the second SPP value which configures the multi-channel filter, or ii) in response to detecting a presence of playback reference signals the first DNN is selected to produce the first SPP value which configures the multi-channel filter. ### Claim 16 (depends on 13) The system of claim 13 further comprising a first loudspeaker cabinet housing in which the processor and the memory are integrated, wherein the multi-channel speech pickup comprises a first plurality of acoustic microphone signals produced by a plurality of acoustic microphones, respectively, that are integrated in the first loudspeaker cabinet housing. ### Claim 17 (depends on 16) The system of claim 16 wherein the acoustic echo cancellation receives as input i) a first plurality of playback reference signals that represent inputs to a first plurality of speaker drivers that are in the first loudspeaker cabinet housing, and ii) a second plurality of playback reference signals that represent inputs to a second plurality of speaker drivers that are in a second loudspeaker cabinet housing. ### Claim 18 (depends on 12) The system of claim 12 further comprising a first loudspeaker cabinet housing in which the processor and the memory are integrated, wherein the multi-channel speech pickup comprises a plurality of acoustic microphone signals produced by a first plurality of acoustic microphones, respectively, that are integrated in the first loudspeaker cabinet housing. ### Claim 19 (depends on 18) The system of claim 18 further comprising a second loudspeaker cabinet having a second plurality of acoustic microphones, an acoustic echo canceller coupled to the second plurality of acoustic microphones and configured to produce a plurality of linear echo estimates, and a wireless transmitter that is to transmit second side information from the plurality of linear echo estimates to the first loudspeaker cabinet, wherein the extracted first plurality of features being input to the first DNN also include a plurality of features extracted from the second side information. ### Claim 20 A method for performing speech signal enhancement, comprising: performing acoustic echo cancellation upon a plurality of microphone signals, to produce i) a multi-channel speech pickup as a plurality of echo cancelled audio signals and ii) a plurality of linear echo estimates associated with the echo cancelled audio signals, extracting a plurality of features from i) a current frame of the multi-channel speech pickup and ii) side information selected from the plurality of linear echo estimates, a diffuse signal component, or a noise estimate of the multi-channel speech pickup; producing speech presence probability value, SPP value, for the current frame, wherein the SPP value is produced based on the extracted plurality of features; and applying the SPP value to configure a multi-channel filter whose input is the multi-channel speech pickup and whose output is a single audio signal. ### Claim 21 (depends on 20) The method of claim 20 wherein the multi-channel filter is configured to perform multi-channel residual echo suppression upon the multi-channel speech pickup.