# US 2020/0395035 A1 · Automatic speech recognition system addressing perceptual-based adversarial audio attacks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0395035 A1 - Document ID: US20200395035A1 - Status: Published application - Kind: A1 - Application number: 16/897,673 - Filed: 2020-06-10 - Published: 2020-12-17 - Earliest claimed: 2019-06-14 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Jeremy KOLTER; Joseph SZURLEY - Web page: https://patentgrep.com/patent/US20200395035A1 ## CPC - G10L 15/01: Assessment or evaluation of speech recognition systems - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L 15/10: using distance or distortion measures between unknown speech and reference templates - G10L 15/12: using dynamic programming techniques, e.g. dynamic time warping [DTW] - G10L 25/45: characterised by the type of analysis window - G10L 19/018: Audio watermarking, i.e. embedding inaudible data in the audio signal - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A computer-implemented method for creating a combined audio signal in a speech recognition system, the method includes sampling the audio input signal to generate a time-domain sampled input signal, then converting the time-domain sampled input signal to a frequency-domain input signal, afterwards generating perceptual weights in response to frequency components of critical bands of the frequency-domain input signal, creating a time-domain adversary signal in response to the perceptual weights; and combining the time-domain adversary signal with the audio input signal to create a combined audio signal, wherein a speech processing of the combined audio signal will output a different result from speech processing of the audio input signal. ## Claims ### Claim 1 A computer-implemented method for speech recognition, the method comprising: sampling an audio input signal to generate a time-domain sampled input signal; converting the time-domain sampled input signal to a frequency-domain input signal; generating perceptual weights in response to frequency components of critical bands of the frequency-domain input signal; creating a time-domain adversary signal in response to the perceptual weights; and combining the time-domain adversary signal with the audio input signal to create a combined audio signal, wherein a speech processing of the combined audio signal outputs a different result from speech processing of the audio input signal. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the time-domain adversary signal is a combination of a perceptual based loss signal and an adversarial loss signal. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the combination of the perceptual based loss signal and the adversarial loss signal are represented by an equation min | | δ | | 2  ≤ ∈  α  L  ( x + δ , y target ) + ( 1 - α )  L percep  ( δ ) ≡ min | | δ | | 2  ≤ ∈  L t  o  t  ( x + δ , y target , α ) , where δ is a norm constraint. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the norm constraint δ is an l2 norm constraint and the combination of the perceptual based loss and the adversarial loss is solved via a projected gradient descent (PGD) method. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the time-domain sampled input signal is converted to the frequency-domain input signal via a symmetrical algorithm including a Fast Fourier Transform, a Discrete Fourier Transform, or a symmetrical filter bank. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein sampling the audio input signal is via an analog to digital A/D converter, and outputting the time-domain adversary is directly to a digital to analog D/A converter. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 further comprising identifying the critical bands via a psychoacoustic model of a human ear. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the psychoacoustic model is an MPEG psychoacoustic model or an AAC psychoacoustic model. ### Claim 9 A non-transitory computer-readable medium tangibly embodying computer readable instructions for a software program, the software program being executable by a processor of a computing device to provide operations comprising: sampling an audio input signal to generate a time-domain sampled input signal; converting the time-domain sampled input signal to a frequency-domain input signal; generating perceptual weights in response to frequency components of critical bands of the frequency-domain input signal; creating a time-domain adversary signal in response to the perceptual weights; and combining the time-domain adversary signal with the audio input signal to create a combined audio signal. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the software program is executable by the processor of the computing device to provide a further operation of identifying the critical bands via a psychoacoustic model of a human ear. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the time-domain adversary signal is a combination of a perceptual based loss signal and an adversarial loss signal. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the combination of the perceptual based loss and the adversarial loss are represented by an equation min | | δ | | 2  ≤ ∈  α  L  ( x + δ , y target ) + ( 1 - α )  L percep  ( δ ) ≡ min | | δ | | 2  ≤ ∈  L t  o  t  ( x + δ , y target , α ) . ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein sampling the audio input signal is via an analog to digital A/D converter, and outputting the time-domain adversary signal is directly to a digital to analog D/A converter. ### Claim 14 A computer system for performing adversarial audio attacks to an automated speech recognition system having a processor configured to execute computer-readable instructions and a memory configured to execute the computer-executable instructions, the computer-executable instructions when executed by the processor perform the following functions: sampling an audio input signal to generate a time-domain sampled input signal; converting the time-domain sampled input signal to a frequency-domain input signal; identifying the critical bands via a psychoacoustic model of the human ear; generating perceptual weights based on frequency components of the critical bands of the frequency-domain input signal; creating a time-domain adversary signal based on the perceptual weights; and combining the time-domain adversary signal with the audio input signal to create a combined audio signal. ### Claim 15 (depends on 14) The computer system of claim 14, wherein the time-domain adversary signal is a combination of a perceptual based loss signal and an adversarial loss signal. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the combination of the perceptual based loss signal and the adversarial loss signal are represented by an equation min | | δ | | 2  ≤ ∈  α  L  ( x + δ , y target ) + ( 1 - α )  L percep  ( δ ) ≡ min | | δ | | 2  ≤ ∈  L t  o  t  ( x + δ , y target , α ) . ### Claim 17 (depends on 16) The computer system of claim 16, wherein sampling the audio input signal is via an analog to digital A/D converter, and outputting the time-domain adversary signal is directly to a digital to analog D/A converter. ### Claim 18 (depends on 14) The computer system of claim 14, wherein the computer-executable instructions when executed by the processor perform the further function of identifying the critical bands via a psychoacoustic model of a human ear. ### Claim 19 (depends on 18) The computer system of claim 18, wherein the psychoacoustic model is an MPEG psychoacoustic model or an AAC psychoacoustic model. ### Claim 20 (depends on 15) The computer system of claim 15, wherein a speech processing of the combined audio signal outputs a different result from speech processing of the audio input signal.