On-device personalization of speech synthesis for training of speech recognition model(s)

A client device comprising:

• at least one processor;
• and
• memory storing instructions that, when executed by the at least one processor, cause the at least one processor to:
• identify a textual segment stored locally at the client device of a user;
• process, using a trained on-device text-to-speech (US) model stored locally in local storage the client device, the textual segment to generate synthesized speech audio data that includes synthesized speech corresponding to the textual segment;
• identify ground truth audio data that captures a given spoken utterance of the user, the textual segment being a ground truth textual segment for the ground truth audio data;
• generate a gradient based on comparing the synthesized speech audio data and the ground truth audio data;
• and
• update local weights of the on-device TTS model based on the generated gradient.

A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a client device, cause the at least one processor of the client device to perform operations, the operations comprising:

• identifying a textual segment stored locally at the client device of a user;
• processing, using a trained on-device text-to-speech (TTS) model stored locally in local storage the client device, the textual segment to generate synthesized speech audio data that includes synthesized speech corresponding to the textual segment;
• identifying ground truth audio data that captures a given spoken utterance of the user, the textual segment being a ground truth textual segment for the ground truth audio data;
• generating a gradient based on comparing the synthesized speech audio data and the ground truth audio data;
• and
• updating local weights of the on-device TTS model based on the generated gradient.