Systems and methods of processing audio data with a multi-rate learnable audio frontend

An audio processing system comprising:

• a processor;
• and
• memory having instructions that, when executed by the processor, cause the processor to
• receive a one-dimensional audio waveform;
• process the one-dimensional waveform via a multi-stage learnable audio frontend model to convert the one-dimensional audio waveform into a two-dimensional matrix representing features of the audio waveform, wherein the multi-stage learnable audio frontend model is configured to:
• apply a first filterbank h(0) to the audio waveform to generate a first time-frequency representation f(0) of the audio waveform;
• apply a first decimation filter d(0) to the audio waveform to generate a first decimated audio input y(1);
• apply a second filterbank h(1) to the first decimated audio input y(1) to generate a second time-frequency representation f(1) of the audio waveform;
• and
• stack the first time-frequency representation f(0) and the second time-frequency representation f(1) together to generate the two-dimensional matrix f;
• and
• process the two-dimensional matrix using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a respective output for each of one or more audio understanding tasks.

A method of processing audio data with a multi-stage learnable audio frontend model, the method comprising:

• receiving, as input, a one-dimensional audio waveform y(0);
• processing the audio waveform y(0) using a multi-stage learnable audio frontend model to convert the one-dimensional audio waveform into a two-dimensional matrix representing features of the audio waveform, wherein the multi-stage learnable audio frontend model is configured to:
• apply a first filterbank h(0) to the audio waveform to generate a first time-frequency representation f(0) of the audio waveform;
• apply a first decimation filter d(0) to the audio waveform to generate a first decimated audio input y(1);
• apply a second filterbank h(1) to the first decimated audio input y(1) to generate a second time-frequency representation f(1) of the audio waveform;
• apply a second decimation filter d(1) to the first decimated audio input y(1) to generate a second decimated audio input y(2);
• apply a third filterbank h(2) to the second decimated audio input y(2) to generate a third time-frequency representation f(2) of the audio waveform;
• and
• stack the first time-frequency representation f(0), the second time-frequency representation f(1), and the third time-frequency representation f(2) together to generate the two-dimensional matrix f;
• and
• processing the two-dimensional matrix using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a respective output for each of one or more audio understanding tasks.