# US 2022/0284891 A1 · Noisy student teacher training for robust keyword spotting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284891 A1
- Document ID: US20220284891A1
- Status: Published application
- Kind: A1
- Application number: 17/190,779
- Filed: 2021-03-03
- Published: 2022-09-08
- Earliest claimed: 2021-03-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Hyun Jin Park; Pai Zhu; Ignacio Lopez Moreno; Niranjan Subrahmanya
- Web page: https://patentgrep.com/patent/US20220284891A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 18/24: Classification techniques
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/096: Transfer learning
- G10L 15/063: {Training}
- G10L 15/08: Speech classification or search
- G10L 21/0208: Noise filtering
- G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence)
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2021/02082: {the noise being echo, reverberation of the speech}
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Teacher-student learning can be used to train a keyword spotting (KWS) model using augmented training instance(s). Various implementations include aggressively augmenting (e.g., using spectral augmentation) base audio data to generate augmented audio data, where one or more portions of the base instance of audio data can be masked in the augmented instance of audio data (e.g., one or more time frames can be masked, one or more frequencies can be masked, etc.). Many implementations include processing augmented audio data using a KWS teacher model to generate a soft label, and processing the augmented audio data using a KWS student model to generate predicted output. One or more portions of the KWS student model can be updated based on a comparison of the soft label and the generated predicted output.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
training an initial keyword spotting (“KWS”) teacher model using a labeled training data set which includes input audio features and supervised output features;
generating augmented audio data, wherein generating the augmented audio data comprises augmenting an instance of base audio data;
processing the augmented audio data using the initial KWS teacher model to generate a soft label;
processing the augmented audio data using a KWS student model to generate student output; and
updating one or more portions of the KWS student model based on comparing the soft label and the generated student output.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating additional augmented audio data, wherein generating the additional augmented audio data comprises augmenting an additional instance of base audio data;
processing the additional augmented audio data using the initial KWS teacher model to generate an additional soft label;
processing the additional augmented audio data using the KWS student model to generate additional student output; and
further updating the one or more portions of the KWS student model based on comparing the additional soft label and the generated additional student output.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
subsequent to updating the one or more portions of the student model based on comparing the soft label and the generated student output, using the KWS student model as a next instance of the KWS teacher model;
generating additional augmented audio data, wherein generating the additional augmented audio data comprises augmenting an additional instance of base audio data;
processing the additional augmented audio data using the next instance of the KWS teacher model to generate an additional soft label;
processing the additional augmented audio data using a next instance of the KWS student model to generate additional student output; and
updating one or more additional portions of the next instance of the KWS student model based on comparing the additional soft label and the additional student output.

### Claim 4 (depends on 1)

The method of claim 1, wherein augmenting the instance of base audio data comprises:
aggressively augmenting the instance of base audio data.

### Claim 5 (depends on 4)

The method of claim 4, wherein aggressively augmenting the instance of audio data comprises:
processing the instance of base audio data using spectral augmentation to generate the augmented audio data, wherein the spectral augmentation includes time masking of the base audio data and/or frequency masking of the base audio data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the base audio data includes a true occurrence of a keyword for which the initial KWS teach model is trained to predict, and wherein the augmented audio data fails to include the true occurrence of the keyword.

### Claim 7 (depends on 1)

The method of claim 1,
wherein comparing the soft label and the generated student output comprises:
generating a cross-entropy loss based on the soft label and the generated student output; and
wherein updating the one or more portions of the KWS student model based on comparing the soft label and the generated student output comprises:
updating the one or more portions of the KWS student model based on the cross-entropy loss.

### Claim 8 (depends on 1)

The method of claim 1, wherein the instance of base audio data is a labeled instance of base audio data, and wherein processing the augmented audio data using the initial KWS teacher model to generate the soft label comprises:
processing the augmented audio data generated by augmenting the labeled instance of base audio data using the initial KWS teacher model to generate the soft label.

### Claim 9 (depends on 1)

The method of claim 1, wherein the instance of base audio data is an unlabeled instance of base audio data, and wherein processing the augmented audio data using the initial KWS teacher model to generate the soft label comprises:
processing the augmented audio data generated by augmenting the unlabeled instance of base audio data using the initial KWS teacher model to generate the soft label.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving audio data capturing a spoken utterance which includes one or more keywords;
processing the audio data using the KWS student model to generate keyword output; and
determining the one or more keywords are present in the spoken utterance based on the keyword output.

### Claim 11 (depends on 10)

The method of claim 10, wherein the keyword output is binary classification output.

### Claim 12

A method implemented by one or more processors, the method comprising:
receiving audio data capturing a spoken utterance which includes one or more keywords;
processing the audio data using a keyword spotting (“KWS”) model to generate keyword output, wherein training the KWS model comprises:
training an initial KWS teacher model portion of the KWS model using a labeled training data set which includes input audio features and supervised output features;
generating augmented audio data, wherein generating the augmented audio data comprises augmenting an instance of base audio data;
processing the augmented audio data using the initial KWS teacher model to generate a soft label;
processing the augmented audio data using a KWS student model portion of the KWS model to generate student output; and
updating one or more portions of the KWS student model portion of the KWS model based on comparing the soft label and the generated student output; and
determining whether the one or more keywords are present in the spoken utterance based on the keyword output.

### Claim 13

A method implemented by one or more processors, the method comprising:
training an initial teacher model using a labeled training data set which includes input audio features and supervised output features;
generating augmented audio data, wherein generating the augmented audio data comprises augmenting an instance of base audio data using time masking and/or frequency masking of the base audio data;
processing the augmented audio data using the initial teacher model to generate a soft label;
processing the augmented audio data using a student model to generate student output; and
updating one or more portions of the student model based on comparing the soft label and the generated student output.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
generating additional augmented audio data, wherein generating the additional augmented audio data comprises augmenting an additional instance of base audio data;
processing the additional augmented audio data using the initial teacher model to generate an additional soft label;
processing the additional augmented audio data using the student model to generate additional student output; and
further updating the one or more portions of the student model based on comparing the additional soft label and the generated additional student output.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
subsequent to updating the one or more portions of the student model based on comparing the soft label and the generated student output, using the student model as a next instance of the teacher model;
generating additional augmented audio data, wherein generating the additional augmented audio data comprises augmenting an additional instance of base audio data;
processing the additional augmented audio data using the next instance of the teacher model to generate an additional soft label;
processing the additional augmented audio data using a next instance of the student model to generate additional student output; and
updating one or more additional portions of the next instance of the student model based on comparing the additional soft label and the additional student output.

### Claim 16 (depends on 13)

The method of claim 13, wherein comparing the soft label and the generated student output comprises:
generating a cross-entropy loss based on the soft label and the generated student output; and
wherein updating the one or more portions of the student model based on comparing the soft label and the generated student output comprises:
updating the one or more portions of the student model based on the cross-entropy loss.

### Claim 17 (depends on 13)

The method of claim 13, wherein the instance of base audio data is an labeled instance of base audio data, and wherein processing the augmented audio data using the initial teacher model to generate the soft label comprises:
processing the augmented audio data generated by augmenting the labeled instance of base audio data using the initial teacher model to generate the soft label.

### Claim 18 (depends on 13)

The method of claim 13, wherein the instance of base audio data is an unlabeled instance of base audio data, and wherein processing the augmented audio data using the initial teacher model to generate the soft label comprises:
processing the augmented audio data generated by augmenting the unlabeled instance of base audio data using the initial teacher model to generate the soft label.
