# US 12,217,765 B2 · Robust short-time fourier transform acoustic echo cancellation during audio playback > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,765 B2 - Document ID: US12217765B2 - Status: Granted - Kind: B2 - Application number: 18/313,013 - Filed: 2023-05-05 - Published: 2023-12-07 - Granted: 2025-02-04 - Earliest claimed: 2017-09-27 - Assignees: Sonos, Inc. - Applicants: Sonos, Inc. - Inventors: Daniele Giacobello - Web page: https://patentgrep.com/patent/US12217765B2 ## CPC - G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) - G10K 11/178: by electro-acoustically regenerating the original acoustic waves in anti-phase - G10L 21/0208: Noise filtering - H04M 9/082: {using echo cancellers (echo cancellers per se H04B3/23)} - G10K 2210/3012: Algorithms - G10K 2210/3028: Filtering, e.g. Kalman filters or special analogue or digital filters - G10K 2210/505: Echo cancellation, e.g. multipath-, ghost- or reverberation-cancellation - G10L 21/0232: Processing in the frequency domain - G10L 2021/02087: {the noise being separate speech, e.g. cocktail party} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)} - H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) - H04R 29/007: {for public address systems (public address systems per se H04R27/00)} - H04R 2227/003: Digital PA systems using, e.g. LAN or internet - H04R 2227/005: Audio distribution systems for home, i.e. multi-room use - H04R 2420/03: Connection circuits to selectively connect loudspeakers or headphones to amplifiers - H04R 2420/07: Applications of wireless loudspeakers or wireless microphones - H04R 2430/23: Direction finding using a sum-delay beam-former - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G10K: SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Example techniques involve noise-robust acoustic echo cancellation. An example implementation may involve causing one or more speakers of the playback device to play back audio content and while the audio content is playing back, capturing, via the one or more microphones, audio within an acoustic environment that includes the audio playback. The example implementation may involve determining measured and reference signals in the STFT domain. During each nth iteration of an acoustic echo canceller (AEC): the implementation may involve determining a frame of an output signal by generating a frame of a model signal by passing a frame of the reference signal through an instance of an adaptive filter and then redacting the nth frame of the model signal from an nth frame of the measured signal. The implementation may further involve determining an instance of the adaptive filter for a next iteration of the AEC. ## Claims ### Claim 1 A system comprising: at least one playback device comprising at least one headphone, at least one microphone and one or more audio transducers; a mobile device; one or more wireless network interfaces; at least one processor; and data storage storing program instructions that are executable by the at least one processor to cause the system to perform functions comprising: playing back at least one audio signal via the one or more audio transducers of the at least one playback device; while playing back the at least one audio signal, capturing, via the at least one microphone, audio, wherein at least a portion of the captured audio represents sound produced by the one or more audio transducers in playing back the at least one audio signal; transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo; transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal; during each nth iteration of an acoustic echo canceller (AEC): determining an nth frame of an output signal, wherein determining the nth frame of the output signal comprises: (i) generating an nth frame of a model signal representing estimated acoustic echo by passing an nth frame of the reference signal through an nth instance of an adaptive filter; and (ii) generating the nth frame of the output signal by differencing the nth frame of the model signal and an nth frame of the measured signal; determining a n+1th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1th instance of the adaptive filter for the next iteration of the AEC comprises: (i) estimating an nth frame of an error signal, the nth frame of the error signal representing a difference between the nth frame of the measured signal and the nth frame of the model signal; (ii) converting the nth frame of an error signal to an nth update filter; (iii) deactivating inactive portions of the nth update filter, the inactive portions having less than a threshold energy; and (iv) generating the n+1th instance of the adaptive filter for the next iteration of the AEC by summing the nth instance of the adaptive filter with the nth update filter; and sending, via the one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input. ### Claim 2 (depends on 1) The system of claim 1, wherein the at least one headphone comprises wireless headphones, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer of the wireless headphones, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back the at least one audio signal comprises: playing back the first audio channel via the first audio transducer; and playing back the second audio channel via the second audio transducer. ### Claim 3 (depends on 1) The system of claim 1, wherein the at least one playback device comprises a first playback device and a second playback device configured in a bonded configuration, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back at least one audio signal comprises: playing back the first audio channel via the first audio transducer; and playing back the second audio channel via the second audio transducer. ### Claim 4 (depends on 1) The system of claim 1, wherein the mobile device comprises the AEC, and wherein the functions further comprise: sending, via the one or more wireless network interfaces to the mobile device, data representing the captured audio. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more wireless network interfaces comprise an 802.15-compatible wireless network interface, and wherein the functions further comprise: sending, via the 802.15-compatible wireless network interface to the at least one playback device, data representing the at least one audio signal. ### Claim 6 (depends on 1) The system of claim 1, wherein converting the nth frame of an error signal to the nth update filter comprises determining a normalized least mean square (NMLS) of the nth frame of the error signal, and wherein deactivating the inactive portions of the nth update filter comprises determining a sparse NMLS of the nth frame of the error signal by applying to the NMLS of the nth frame of the error signal, a sparse partition criterion that zeroes out frequency bands of the NMLS having less than the threshold energy. ### Claim 7 (depends on 6) The system of claim 6, wherein determining the normalized least mean square (NMLS) of the nth frame of the error signal comprises applying a frequency-dependent regularization parameter to adapt an NMLS learning rate of change between AEC iterations according to a magnitude of the measured signal. ### Claim 8 (depends on 6) The system of claim 6, wherein converting the sparse NMLS of the nth frame of the error signal to the nth update filter comprises: converting the sparse NMLS of the nth frame to a matrix of filter coefficients; and cross-band filtering the matrix of filter coefficients to generate the nth update filter. ### Claim 9 (depends on 6) The system of claim 6, wherein the data storage further comprises program instructions that are executable by the at least one processor such that the system is configured to perform functions further comprising: before determining the NMLS of the nth frame of the error signal, applying an error recovery non-linearity function to the error signal to limit the error signal to a threshold magnitude, wherein determining the normalized least mean square (NMLS) of the nth frame of the error signal comprises determining the NMLS of the nth frame of the limited error signal. ### Claim 10 (depends on 9) The system of claim 9, wherein the error recovery non-linearity function comprises a non-linear clipping function that limits portions of the error signal that are above the threshold magnitude to the threshold magnitude. ### Claim 11 A mobile device comprising: one or more wireless network interfaces; at least one processor; and data storage storing program instructions that are executable by the at least one processor to cause the mobile device to perform functions comprising: causing at least one playback device to play back at least one audio signal via one or more audio transducers of the at least one playback device, wherein the at least one playback device comprises at least one headphone; while the at least one playback device is playing back the at least one audio signal, receiving audio captured via at least one microphone of the at least one playback device, wherein at least a portion of the captured audio represents sound produced by one or more audio transducers of the at least one playback device in playing back the at least one audio signal; transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo; transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal; during each nth iteration of an acoustic echo canceller (AEC): determining an nth frame of an output signal, wherein determining the nth frame of the output signal comprises: (i) generating an nth frame of a model signal representing estimated acoustic echo by passing an nth frame of the reference signal through an nth instance of an adaptive filter; and (ii) generating the nth frame of the output signal by differencing the nth frame of the model signal and an nth frame of the measured signal; determining a n+1th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1th instance of the adaptive filter for the next iteration of the AEC comprises: (i) estimating an nth frame of an error signal, the nth frame of the error signal representing a difference between the nth frame of the measured signal and the nth frame of the model signal; (ii) converting the nth frame of an error signal to an nth update filter; (iii) deactivating inactive portions of the nth update filter, the inactive portions having less than a threshold energy; and (iv) generating the n+1th instance of the adaptive filter for the next iteration of the AEC by summing the nth instance of the adaptive filter with the nth update filter; and sending, via the one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input. ### Claim 12 (depends on 11) The mobile device of claim 11, wherein the at least one headphone comprise wireless headphones, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer of the wireless headphones, and wherein causing the at least one playback device to play back the at least one audio signal comprises: causing the wireless headphones to play back the at least one audio signal via at the first audio transducer and the second audio transducer. ### Claim 13 (depends on 11) The mobile device of claim 11, wherein the at least one playback device comprises a first playback device and a second playback device configured in a bonded configuration, wherein the one or more audio transducers comprise a first audio transducer and a second audio transducer, and wherein the at least one audio signal comprises a first audio channel and a second audio channel, and wherein playing back the at least one audio signal comprises: causing the first playback device to play back the first audio channel via the first audio transducer; and causing the second playback device to play back the second audio channel via the second audio transducer. ### Claim 14 (depends on 11) The mobile device of claim 11, wherein the one or more wireless network interfaces comprise an 802.15-compatible wireless network interface, and wherein the functions further comprise: sending, via the 802.15-compatible wireless network interface to the at least one playback device, data representing the at least one audio signal. ### Claim 15 (depends on 11) The mobile device of claim 11, wherein converting the nth frame of an error signal to the nth update filter comprises determining a normalized least mean square (NMLS) of the nth frame of the error signal, and wherein deactivating inactive portions of the nth update filter comprises determining a sparse NMLS of the nth frame of the error signal by applying to the NMLS of the nth frame of the error signal, a sparse partition criterion that zeroes out frequency bands of the NMLS having less than the threshold energy. ### Claim 16 (depends on 15) The mobile device of claim 15, wherein determining the normalized least mean square (NMLS) of the nth frame of the error signal comprises applying a frequency-dependent regularization parameter to adapt an NMLS learning rate of change between AEC iterations according to a magnitude of the measured signal. ### Claim 17 (depends on 15) The mobile device of claim 15, wherein converting the sparse NMLS of the nth frame of the error signal to the nth update filter comprises: converting the sparse NMLS of the nth frame to a matrix of filter coefficients; and cross-band filtering the matrix of filter coefficients to generate the nth update filter. ### Claim 18 (depends on 15) The mobile device of claim 15, wherein the data storage further comprises program instructions that are executable by the at least one processor such that the mobile device is configured to perform functions further comprising: before determining the NMLS of the nth frame of the error signal, applying an error recovery non-linearity function to the error signal to limit the error signal to a threshold magnitude, wherein determining the normalized least mean square (NMLS) of the nth frame of the error signal comprises determining the NMLS of the nth frame of the limited error signal. ### Claim 19 (depends on 18) The mobile device of claim 18, wherein the error recovery non-linearity function comprises a non-linear clipping function that limits portions of the error signal that are above the threshold magnitude to the threshold magnitude. ### Claim 20 At least one non-transitory computer-readable medium comprising program instructions that are executable by at least one processor such that a system is configured to perform functions comprising: causing at least one playback device to play back at least one audio signal via one or more audio transducers of the at least one playback device, wherein the at least one playback device comprises at least one headphone; while the at least one playback device is playing back the at least one audio signal, receiving audio captured via at least one microphone of the at least one playback device, wherein at least a portion of the captured audio represents sound produced by the one or more audio transducers of the at least one playback device in playing back the at least one audio signal; transforming into a short time Fourier transform (STFT) domain the captured audio to generate a measured signal representing actual acoustic echo; transforming into the STFT domain the at least one audio signal being played back by the at least one playback device to generate a reference signal; during each nth iteration of an acoustic echo canceller (AEC): determining an nth frame of an output signal, wherein determining the nth frame of the output signal comprises: (i) generating an nth frame of a model signal representing estimated acoustic echo by passing an nth frame of the reference signal through an nth instance of an adaptive filter; and (ii) generating the nth frame of the output signal by differencing the nth frame of the model signal and an nth frame of the measured signal; determining a n+1th instance of the adaptive filter for a next iteration of the AEC, wherein determining the n+1th instance of the adaptive filter for the next iteration of the AEC comprises: (i) estimating an nth frame of an error signal, the nth frame of the error signal representing a difference between the nth frame of the measured signal and the nth frame of the model signal; (ii) converting the nth frame of an error signal to an nth update filter; (iii) deactivating inactive portions of the nth update filter, the inactive portions having less than a threshold energy; and (iv) generating the n+1th instance of the adaptive filter for the next iteration of the AEC by summing the nth instance of the adaptive filter with the nth update filter; and sending, via one or more wireless network interfaces, the output signal as a voice input to one or more voice assistants for processing of the voice input.