# US 12,475,906 B2 · Method and system for multiple time resolution audio processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,475,906 B2 - Document ID: US12475906B2 - Status: Granted - Kind: B2 - Application number: 18/450,784 - Filed: 2023-08-16 - Published: 2025-02-20 - Granted: 2025-11-18 - Earliest claimed: 2023-08-16 - Assignees: Analog Devices, Inc. - Applicants: Analog Devices, Inc. - Inventors: Johannes Traa; Atulya Yellepeddi; Donald F. Porges - Web page: https://patentgrep.com/patent/US12475906B2 ## CPC - G10L 21/0224: Processing in the time domain - G10L 21/0232: Processing in the frequency domain - G10L 21/0208: Noise filtering - G10L 21/0216: characterised by the method used for estimating noise - G10L 15/00: Speech recognition (G10L17/00 takes precedence) - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2021/02166: {Microphone arrays; Beamforming} - G10L 2021/02168: {the estimation exclusively taking place during speech pauses} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Aspects of the present disclosure provided a method for voice control that includes transforming, using a short-time Fourier transform (STFT) applied to data in each window aligned across each input channel of the multichannel audio stream, the multichannel audio stream into a complex valued frequency-domain representation. For a current window, the method further includes: updating a first complex-valued covariance matrix corresponding to a slowly-adapting beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing a magnitude of the single-channel denoised estimate to a magnitude of each input channel of the multichannel audio stream; and selectively updating or refraining from updating, responsive to the VAD estimate respectively indicating a presence or an absence of speech, a second complex-valued covariance matrix corresponding to a quickly-adapting beamformer. ## Claims ### Claim 1 A method for voice control, comprising: transforming, using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of a multichannel audio stream, the multichannel audio stream into a complex valued frequency-domain representation, wherein for a current one of the plurality of windows, the method comprises: updating a first complex-valued covariance matrix corresponding to a slowly-adapting beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing a magnitude of the single-channel denoised estimate to a magnitude of each input channel of the multichannel audio stream; and selectively updating or refraining from updating, responsive to the VAD estimate respectively indicating a presence or an absence of speech, a second complex-valued covariance matrix corresponding to a quickly-adapting beamformer; and controlling, by a hardware processor, a voice user interface based device to perform a user perceptible action, responsive to an output of at least the quickly-adapting beamformer. ### Claim 2 (depends on 1) The method in accordance with claim 1, further comprising configuring the slowly-adapting beamformer and the quickly-adapting beamformer to use different equations for an update rule and different adaptation speeds. ### Claim 3 (depends on 1) The method in accordance with claim 1, further comprising configuring the slowly-adapting beamformer and the quickly-adapting beamformer to both use a same equation for an update rule, but have different adaptation speeds. ### Claim 4 (depends on 3) The method in accordance with claim 3, further comprising: configuring the slowly-adapting beamformer to slowly adapt to an acoustic background including background noise in the multichannel audio stream; and configuring the quickly-adapting beamformer to quickly adapt to speech in the multichannel audio stream. ### Claim 5 (depends on 1) The method in accordance with claim 1, wherein the method further comprises, responsive to the magnitude of the single-channel denoised estimate being lower than the magnitude of a given input channel of the multichannel audio stream by a threshold amount, considering a particular frequency band in the STFT corresponding to the single-channel denoised estimate to be noise and selectively refraining from updating the second complex-valued covariance matrix corresponding to the quickly-adapting beamformer. ### Claim 6 (depends on 1) The method in accordance with claim 1, further comprising applying the method in a beamforming strategy configured to use a correlation matrix of any of the multichannel audio stream and noise. ### Claim 7 (depends on 1) The method in accordance with claim 1, wherein forming the single-channel denoised estimate comprises calculating a beamformer vector at each frequency band in the STFT. ### Claim 8 (depends on 7) The method in accordance with claim 7, wherein the beamformer vector is calculated at each frequency band in the STFT responsive to the frequency-domain representation of the current one of the plurality of windows. ### Claim 9 (depends on 7) The method in accordance with claim 7, wherein the single-charmed denoised estimate is determined by multiplying the beamformer vector by a corresponding beamformer weight vector of the STFT. ### Claim 10 (depends on 1) The method in accordance with claim 1, wherein forming the single-channel denoised estimate comprises calculating a minimum variance distortionless response (MVDR) beamformer vector at each frequency band in the STFT. ### Claim 11 (depends on 1) The method in accordance with claim 1, wherein forming the single-channel denoised estimate comprises calculating a minimum power distortionless response (MPDR) beamformer vector at each frequency band in the STFT. ### Claim 12 (depends on 1) The method in accordance with claim 1, further comprising receiving the multichannel audio stream from a microphone array and a desired source direction-of-arrival (DOA). ### Claim 13 (depends on 1) The method in accordance with claim 1, wherein the transforming comprises applying frequency-domain processing to the complex valued frequency-domain representation to obtain processed audio, and the method further includes using an inverse STFT and an overlap-add procedure to transform the processed audio from a frequency domain to a time domain and provide a time-domain representation. ### Claim 14 (depends on 13) The method in accordance with claim 13, further comprising performing the method on-line using a first buffer configured to collect input samples and a second buffer configured to feed out the processed audio. ### Claim 15 (depends on 1) The method in accordance with claim 1, further comprising performing the method by an automatic speech recognition system operatively coupled to a voice user interface of the voice user interface based device and configured to convert the speech to commands to control the voice user interface based device to perform the user perceptible action. ### Claim 16 (depends on 1) The method in accordance with claim 1, wherein the plurality of windows are exponential windows. ### Claim 17 (depends on 1) The method in accordance with claim 1, wherein the plurality of windows are rectangular windows of different lengths, wherein the slowly-adapting beamformer uses a time window having a longer length than the quickly-adapting beamformer to capture a stationarity of time. ### Claim 18 (depends on 1) The method in accordance with claim 1, wherein the method is performed off-line. ### Claim 19 (depends on 1) The method in accordance with claim 1, further comprising writing a covariance update step and beamformer vector calculation step in a mathematically equivalent form to avoid computing matrix inversions and limit to algebraic operations. ### Claim 20 (depends on 1) The method in accordance with claim 1, further comprising weighting speech captured from a driver's location more heavily than speech captured from a passenger's location. ### Claim 21 (depends on 1) The method in accordance with claim 1, further comprising weighting speech captured from a front seat passenger's location more heavily than speech captured from a back seat passenger's location. ### Claim 22 A computer program product configured to enable voice control, the computer program product comprising one or more non-transitory computer-readable media, having instructions stored thereon that when executed by one or more processors cause the one or more processors, individually or in combination, to perform a method comprising: transforming, using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of a multichannel audio stream, the multichannel audio stream into a complex valued frequency-domain representation, wherein for a current one of the plurality of windows, the method comprises: updating a first complex-valued covariance matrix corresponding to a slowly-adapting beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing a magnitude of the single-channel denoised estimate to a magnitude of each input channel of the multichannel audio stream; and selectively updating or refraining from updating, responsive to the VAD estimate respectively indicating a presence or an absence of speech, a second complex-valued covariance matrix corresponding to a quickly-adapting beamformer; and controlling, by a hardware processor, a voice user interface based device to perform a user perceptible action, responsive to an output of at least the quickly-adapting beamformer. ### Claim 23 A voice control system, comprising: one or more memories, individually or in combination, having instructions; one or more processors each coupled to at least one of the one or more memories and configurable to execute the instructions to transform, using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of a multichannel audio stream, the multichannel audio stream into a complex valued frequency-domain representation, wherein for a current one of the plurality of windows, the one or more processors are configurable to execute the instructions to: update a first complex-valued covariance matrix corresponding to a slowly-adapting beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; calculate a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing a magnitude of the single-channel denoised estimate to a magnitude of each input channel of the multichannel audio stream; and selectively update or refraining from updating, responsive to the VAD estimate respectively indicating a presence or an absence of speech, a second complex-valued covariance matrix corresponding to a quickly-adapting beamformer; and control a voice user interface based device to perform a user perceptible action, responsive to an output of at least the quickly-adapting beamformer.