Multi-domain fine-tuning for contextual adapters of automatic speech recognition

A method, comprising:

• receiving, at a speech recognition system, a plurality of entities in a domain to adapt an automatic speech recognition system;
• and
• executing, by the speech recognition system, an automatic speech recognition pipeline on given audio data to generate a text for speech in the audio data, wherein the automatic speech recognition pipeline comprises:
• a fine-tuned version of a general speech recognition machine learning model;
• and
• a fine-tuned version of contextual adapter for the domain;
• wherein the fine-tuned version of the general speech recognition machine learning model generates an initial version of the text provided to the fine-tuned version of the contextual adapter for the domain and final version of the text based on a respective attention scores for the plurality of entities received from the fine-tuned version of the contextual adapter for the domain;
• and
• wherein the fine-tuned version of the general speech recognition machine learning model and the fine-tuned version of the contextual adapter were fine-tuned in stages, wherein an initial tuning stage fine-tuned the general speech recognition machine learning model alone on first training data comprising a plurality of domains, including the domain, and wherein subsequent tuning stages fine-tuned the contextual adapter while freezing the general speech recognition machine learning model and fine-tuned the general speech recognition machine learning model again.

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

• receiving a plurality of entities in a domain to adapt an automatic speech recognition system;
• and
• causing execution of an automatic speech recognition pipeline on given audio data to generate a text for speech in the audio data, wherein the automatic speech recognition pipeline comprises:
• a fine-tuned version of a general speech recognition machine learning model;
• and
• a fine-tuned version of contextual adapter for the domain;
• wherein the fine-tuned version of the general speech recognition machine learning model generates an initial version of the text provided to the fine-tuned version of the contextual adapter for the domain and final version of the text based on a respective attention scores for the plurality of entities received from the fine-tuned version of the contextual adapter for the domain;
• and
• wherein the fine-tuned version of the general speech recognition machine learning model and the fine-tuned version of the contextual adapter were fine-tuned in stages, wherein an initial tuning stage fine-tuned the general speech recognition machine learning model alone on first training data comprising a plurality of domains, including the domain, and wherein subsequent tuning stages fine-tuned the contextual adapter while freezing the general speech recognition machine learning model and fine-tuned the general speech recognition machine learning model again.