# US 12,112,741 B2 · System and method for data augmentation and speech processing in dynamic acoustic environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,741 B2 - Document ID: US12112741B2 - Status: Granted - Kind: B2 - Application number: 17/178,686 - Filed: 2021-02-18 - Published: 2022-08-18 - Granted: 2024-10-08 - Earliest claimed: 2021-02-18 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Patrick A. Naylor; Dushyant Sharma; Uwe Helmut Jost; William F Ganong, III - Web page: https://patentgrep.com/patent/US12112741B2 ## CPC - G10L 15/063: {Training} - G10L 21/007: characterised by the process used - G06N 20/00: Machine learning - G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence) - G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) - G10L 2021/02166: {Microphone arrays; Beamforming} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method, computer program product, and computing system for defining a model representative of a plurality of acoustic variations to a speech signal, thus defining a plurality of time-varying spectral modifications. The plurality of time-varying spectral modifications may be applied to a reference signal using a filtering operation, thus generating a time-varying spectrally-augmented signal. ## Claims ### Claim 1 A computer-implemented method for data augmentation, executed on a computing device, comprising: defining a model representative of a plurality of acoustic variations to a speech signal associated with an adaptive beamforming, thus defining a plurality of time-varying spectral modifications, wherein the plurality of acoustic variations to the speech signal include frequency-based variations in a speech signal beampattern from a movement of a plurality of beampatterns formed by a microphone array configured for the adaptive beamforming and a beamsteering by dynamically modifying and steering the plurality of beampatterns toward a speaker; and applying the plurality of time-varying spectral modifications to a plurality of feature coefficients of a target domain of a reference signal using a filtering operation, thus generating a plurality of time-varying spectrally-augmented feature coefficients of the reference signal, wherein applying the plurality of time-varying spectral modifications to the plurality of feature coefficients of the target domain of the reference signal includes: generating, via a machine learning model, a mapping of the plurality of acoustic variations to one or more feature coefficients of the target domain representative of the frequency-based variations in the speech signal beampattern from the model representative of the plurality of acoustic variations, applying, via the machine learning model, the mapping of the plurality of acoustic variations to the plurality of feature coefficients of the reference signal, and generating, via the machine learning model, augmented data from the reference signal and one or more parameters associated with a particular acoustic variation of the plurality of acoustic variations. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes one or more of: modeling the plurality of acoustic variations to the speech signal as a statistical distribution; and modeling the plurality of acoustic variations to the speech signal as a mathematical model representative of the plurality of acoustic variations associated with a particular use-case scenario. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes receiving one or more inputs associated with the one or more of a speaker location and a speaker orientation relative to a microphone. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein applying the plurality of time-varying spectral modifications to the reference signal using the filtering operation includes one or more of: applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying parameters in time domain filtering; and applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying multiplication factors in frequency domain filtering. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, further comprising: training a speech processing system using the time-varying spectrally-augmented signal and the one or more inputs associated with one or more of the speaker location and the speaker orientation relative to the microphone. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: training a speech processing system using the time-varying spectrally-augmented signal, thus defining a trained speech processing system. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising: performing speech processing via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device. ### Claim 8 A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising: defining a model representative of a plurality of acoustic variations to a speech signal associated with an adaptive beamforming, thus defining a plurality of time-varying spectral modifications, wherein the plurality of acoustic variations to the speech signal include frequency-based variations in a speech signal beampattern from a movement of a plurality of beampatterns formed by a microphone array configured for the adaptive beamforming and a beamsteering by dynamically modifying and steering the plurality of beampatterns toward a speaker; and applying the plurality of time-varying spectral modifications to a plurality of feature coefficients of a target domain of a reference signal using a filtering operation, thus generating a plurality of time-varying spectrally-augmented feature coefficients of the reference signal, wherein applying the plurality of time-varying spectral modifications to the plurality of feature coefficients of the target domain of the reference signal includes: generating, via a machine learning model, a mapping of the plurality of acoustic variations to one or more feature coefficients of the target domain representative of the frequency-based variations in the speech signal beampattern from the model representative of the plurality of acoustic variations, applying, via the machine learning model, the mapping of the plurality of acoustic variations to the plurality of feature coefficients of the reference signal, and generating, via the machine learning model, augmented data from the reference signal and one or more parameters associated with a particular acoustic variation of the plurality of acoustic variations. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes one or more of: modeling the plurality of acoustic variations to the speech signal as a statistical distribution; and modeling the plurality of acoustic variations to the speech signal as a mathematical model representative of the plurality of acoustic variations associated with a particular use-case scenario. ### Claim 10 (depends on 8) The computer program product of claim 8, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes receiving one or more inputs associated with the one or more of a speaker location and a speaker orientation relative to a microphone. ### Claim 11 (depends on 8) The computer program product of claim 8, wherein applying the plurality of time-varying spectral modifications to the reference signal using the filtering operation includes one or more of: applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying parameters in time domain filtering; and applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying multiplication factors in frequency domain filtering. ### Claim 12 (depends on 10) The computer program product of claim 10, further comprising: training a speech processing system using the time-varying spectrally-augmented signal and the one or more inputs associated with one or more of the speaker location and the speaker orientation relative to the microphone. ### Claim 13 (depends on 8) The computer program product of claim 8, further comprising: training a speech processing system using the time-varying spectrally-augmented signal, thus defining a trained speech processing system. ### Claim 14 (depends on 13) The computer program product of claim 13, further comprising: performing speech processing via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device. ### Claim 15 A computing system comprising: a memory; and a processor configured to define a model representative of a plurality of acoustic variations to a speech signal associated with an adaptive beamforming, thus defining a plurality of time-varying spectral modifications, wherein the plurality of acoustic variations to the speech signal include frequency-based variations in a speech signal beampattern from a movement of a plurality of beampatterns formed by a microphone array configured for the adaptive beamforming and a beamsteering by dynamically modifying and steering the plurality of beampatterns toward a speaker, wherein the processor is further configured to apply the plurality of time-varying spectral modifications to a plurality of feature coefficients of a target domain of a reference signal using a filtering operation, thus generating a plurality of time-varying spectrally-augmented feature coefficients of the reference signal, wherein applying the plurality of time-varying spectral modifications to the plurality of feature coefficients of the target domain of the reference signal includes: generating, via a machine learning model, a mapping of the plurality of acoustic variations to one or more feature coefficients of the target domain representative of the frequency-based variations in the speech signal beampattern from the model representative of the plurality of acoustic variations, applying, via the machine learning model, the plurality of time-varying spectral modifications to the plurality of feature coefficients of the reference signal, and generating, via the machine learning model, augmented data from the reference signal and one or more parameters associated with a particular acoustic variation of the plurality of acoustic variations. ### Claim 16 (depends on 15) The computing system of claim 15, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes one or more of: modeling the plurality of acoustic variations to the speech signal as a statistical distribution; and modeling the plurality of acoustic variations to the speech signal as a mathematical model representative of the plurality of acoustic variations associated with a particular use-case scenario. ### Claim 17 (depends on 15) The computing system of claim 15, wherein defining the model representative of the plurality of acoustic variations to the speech signal includes receiving one or more inputs associated with the one or more of a speaker location and a speaker orientation relative to a microphone. ### Claim 18 (depends on 15) The computing system of claim 15, wherein applying the plurality of time-varying spectral modifications to the reference signal using the filtering operation includes one or more of: applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying parameters in time domain filtering; and applying the plurality of time-varying spectral modifications to the reference signal using a plurality of time-varying multiplication factors in frequency domain filtering.