Efficient adaptation of spoken language understanding based on automatic speech recognition using multi-task learning

An electronic device comprising:

• at least one processing device configured to:
• receive, by an automatic speech recognition (ASR)-based spoken language understanding (SLU) model, an input utterance using an audio input device;
• for each token of the input utterance:
• generate, using a shared ASR encoder of the ASR-based SLU model, an acoustic representation of acoustic features of the token, wherein the shared ASR encoder includes a first adapter layer with fine-tuned parameters;
• determine, using an ASR decoder of the ASR-based SLU model, a text representation of the token using the acoustic representation and any previous tokens in the input utterance, wherein the ASR decoder includes a second adapter layer with fine-tuned parameters;
• combine, using a fusion model of the ASR-based SLU model, the text representation and the acoustic representation to generate a joint representation, wherein the joint representation increases performance of an SLU decoder of the ASR-based SLU model, and wherein, to combine the text representation and the acoustic representation to generate the joint representation, the at least one processing device is configured to:
• generate an output by the fusion model by one of:
• using of the acoustic representation as a query and the text representation as a key and value;
• or
• using of the text representation as the query and the acoustic representation as the key and value;
• and
• determine, using the SLU decoder of the ASR-based SLU model, a semantic label associated with the token based on the joint representation and any previous semantic labels;
• and
• perform an action based on the determined semantic labels in the input utterance.

A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to:

• receive, by an automatic speech recognition (ASR)-based spoken language understanding (SLU) model, an input utterance using an audio input device;
• for each token of the input utterance:
• generate, using a shared ASR encoder of the ASR-based SLU model, an acoustic representation of acoustic features of the token, wherein the shared ASR encoder includes a first adapter layer with fine-tuned parameters;
• determine, using an ASR decoder of the ASR-based SLU model, a text representation of the token using the acoustic representation and any previous tokens in the input utterance, wherein the ASR decoder includes a second adapter layer with fine-tuned parameters;
• combine, using a fusion model of the ASR-based SLU model, the text representation and the acoustic representation to generate a joint representation, wherein the instructions that when executed cause the at least one processor to combine the text representation and the acoustic representation to generate the joint representation include:
• instructions that when executed cause the at least one processor to generate an output by the fusion model by one of:
• using of the acoustic representation as a query and the text representation as a key and value;
• or
• using of the text representation as the query and the acoustic representation as the key and value;
• and
• determine, using an SLU decoder of the ASR-based SLU model, a semantic label associated with the token based on the joint representation and any previous semantic labels;
• and
• perform an action based on the determined semantic labels in the input utterance.