Unsupervised federated learning of machine learning model layers

A method performed by one or more processors of a remote system, the method comprising:

• receiving, from a plurality of client devices and at the remote system, gradients, wherein each of the gradients are generated locally at a corresponding one of the plurality of client devices based on:
• unsupervised learning, at the corresponding one of the plurality of client devices, that is based on processing, using a respective local machine learning model stored locally on the corresponding one of the plurality of client devices, audio data that captures a spoken utterance,
• wherein the respective local machine learning model includes a respective first set of local machine learning model layers and a respective second set of machine learning model layers,
• wherein the respective first set of local machine learning model layers is used in generating, based on processing the audio data, an encoding of the audio data,
• wherein the respective second set of local machine learning model layers is used in generating, based on processing the encoding of the audio data that is generated using the respective first set of local machine learning model layers, predicted output, and
• wherein a gradient is generated, using unsupervised learning, based on the predicted output;
• updating, based on the received gradients, weights of global machine learning model layers that are stored remotely at the remote system and that correspond structurally to the respective first set of local machine learning model layers, of the respective local machine learning model, used in generating the encoding of the audio data;
• subsequent to updating the weights of the global machine learning model layers based on the generated gradients:
• generating a combined machine learning model that includes the updated global machine learning model layers and one or more additional layers;
• and
• training the combined machine learning model using supervised learning;
• and
• subsequent to training the combined machine learning model:
• transmitting the combined machine learning model to one or more of the plurality of client devices, wherein the one or more of the plurality of client devices utilize the combined machine learning to make at least one prediction based on further audio data, detected via one or more of the microphones of the client device, that captures at least part of a further spoken utterance of the user of the client device.

A method performed by one or more processors, the method comprising:

• detecting, via one or more microphones of a client device, audio data that captures a spoken utterance of a user of the client device;
• processing, using a local machine learning model stored locally on the client device, the audio data to generate predicted output,
• wherein the local machine learning model includes at least a first set of local machine learning model layers and a second set of machine learning model layers,
• wherein the first set of local machine learning model layers is used in generating, based on processing the audio data, an encoding of the audio data, and
• wherein the second set of local machine learning model layers is used in generating, based on processing the encoding of the audio data that is generated using the first set of local machine learning model layers, the predicted output;
• generating, using unsupervised learning locally at the client device, a gradient based on the predicted output;
• transmitting, to a remote system and from the client device, the generated gradient;
• receiving, at the remote system and from the client device, the generated gradient;
• updating, at the remote system, one or more weights of global machine learning model layers based on the generated gradient, wherein the global machine learning model layers correspond structurally to first set of local machine learning model layers of the local machine learning model;
• subsequent to updating one or more of the weights of the global machine learning model layers based on the generated gradients, training, using supervised learning remotely at the remote system, a combined machine learning model that includes the updated global machine learning model layers and one or more additional layers;
• transmitting, to the client device and from the remote system, the combined machine learning model;
• and
• using the combined machine learning model to make at least one prediction based on further audio data, detected via one or more of the microphones of the client device, that captures at least part of a further spoken utterance of the user of the client device.