Using corrections, of predicted textual segments of spoken utterances, for training of on-device speech recognition model

A method performed by one or more processors of a client device, the method comprising:

• receiving, via one or more microphones of the client device, audio data that captures a spoken utterance of a user of the client device;
• processing, using a speech recognition model that is stored locally at the client device, the audio data to determine a predicted textual segment that is a prediction of the spoken utterance;
• causing at least part of the predicted textual segment to be visually rendered at a display of the client device;
• subsequent to the predicted textual segment being visually rendered:
• receiving, via one or more of the microphones of the client device, additional audio data that captures an additional spoken utterance of the user;
• and
• processing, using the speech recognition model that is stored locally at the client device, the additional audio data to determine an alternate textual segment that corrects the predicted textual segment;
• determining whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device, wherein determining whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device comprises:
• determining a measure of similarity between the predicted textual segment and the alternate textual segment;
• and
• determining that the correction is directed to performance of the speech recognition model that is stored locally at the client device based on the measure of similarity satisfying a threshold;
• and
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device:
• generating a gradient based on comparing at least the predicted textual segment to the alternate textual segment, and
• transmitting, over a network to a remote system, the generated gradient without transmitting any of:
• the audio data, the predicted textual segment, the additional audio data, and the alternate textual segment,
• wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global speech recognition model.

A client device, the client device comprising:

• one or more processors;
• and
• memory storing local instructions that, when executed, cause the one or more processors to be operable to:
• receive, via one or more microphones of the client device, audio data that captures a spoken utterance of a user of the client device;
• process, using a speech recognition model that is stored locally at the client device, the audio data to determine a predicted textual segment that is a prediction of the spoken utterance;
• cause at least part of the predicted textual segment to be visually rendered at a display of the client device;
• subsequent to the predicted textual segment being visually rendered:
• receive, via one or more of the microphones of the client device, additional audio data that captures an additional spoken utterance of the user;
• and
• process, using the speech recognition model that is stored locally at the client device, the additional audio data to determine an alternate textual segment that corrects the predicted textual segment;
• determine whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device, wherein the instructions to determine whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device comprise instructions to:
• determine a measure of similarity between the predicted textual segment and the alternate textual segment;
• and
• determine that the correction is directed to performance of the speech recognition model that is stored locally at the client device based on the measure of similarity satisfying a threshold;
• and
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device:
• generate a gradient based on comparing at least the predicted textual segment to the alternate textual segment, and
• one or both of:
• update one or more weights of the speech recognition model based on the generated gradient;
• or
• transmit, over a network to a remote system, the generated gradient without transmitting any of:
• the audio data, the predicted textual segment, the additional audio data, and the alternate textual segment,
• wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global speech recognition model.

A method performed by one or more processors of a client device, the method comprising:

• receiving, via one or more microphones of the client device, audio data that captures a spoken utterance of a user of the client device;
• processing, using a speech recognition model that is stored locally at the client device, the audio data to determine a predicted textual segment that is a prediction of the spoken utterance;
• causing at least part of the predicted textual segment to be visually rendered at a display of the client device;
• subsequent to the predicted textual segment being visually rendered:
• receiving, via one or more of the microphones of the client device, additional audio data that captures an additional spoken utterance of the user;
• and
• processing, using the speech recognition model that is stored locally at the client device, the additional audio data to determine an alternate textual segment, the alternate textual segment being a correction to the predicted textual segment;
• determining whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device;
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device:
• generating a gradient based on comparing at least the predicted textual segment to the alternate textual segment, and
• one or both of:
• updating one or more weights of the speech recognition model based on the generated gradient;
• or
• transmitting, over a network to a remote system, the generated gradient without transmitting any of:
• the audio data, the predicted textual segment, the additional audio data, and the alternate textual segment,
• wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global speech recognition model;
• and
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is not directed to performance of the speech recognition model that is stored locally at the client device:
• refraining from generating the gradient.

A client device, the client device comprising:

• one or more processors;
• and
• memory storing local instructions that, when executed, cause the one or more processors to be operable to:
• receive, via one or more microphones of the client device, audio data that captures a spoken utterance of a user of the client device;
• process, using a speech recognition model that is stored locally at the client device, the audio data to determine a predicted textual segment that is a prediction of the spoken utterance;
• cause at least part of the predicted textual segment to be visually rendered at a display of the client device;
• subsequent to the predicted textual segment being visually rendered:
• receive, via one or more of the microphones of the client device, additional audio data that captures an additional spoken utterance of the user;
• and
• process, using the speech recognition model that is stored locally at the client device, the additional audio data to determine an alternate textual segment that corrects the predicted textual segment;
• determine whether the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device;
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is directed to performance of the speech recognition model that is stored locally at the client device:
• generate a gradient based on comparing at least the predicted textual segment to the alternate textual segment, and
• one or both of:
• update one or more weights of the speech recognition model based on the generated gradient;
• or
• transmit, over a network to a remote system, the generated gradient without transmitting any of:
• the audio data, the predicted textual segment, the additional audio data, and the alternate textual segment,
• wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global speech recognition model;
• and
• in response to determining that the alternate textual segment that is the correction to the predicted textual segment is not directed to performance of the speech recognition model that is stored locally at the client device:
• refrain from generating the gradient.