Systems and methods for distributed training of deep learning models
AssigneeMovidius Limited
InventorDavid Moloney
Life of this patent
Estimating the term…
Abstract
Systems and methods for distributed training of deep learning models are disclosed. An example local device to train deep learning models includes a reference generator to label input data received at the local device to generate training data, a trainer to train a local deep learning model and to transmit the local deep learning model to a server that is to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and an updater to update the local deep learning model based on the set of weights received from the server.
Claims19
A non-transitory computer readable medium comprising instructions that, when executed, cause a server to at least:
- obtain a first deep learning model from a first device;
- obtain a second deep learning model from a second device;
- determine that first weights of the first deep learning model are associated with a first distribution of weights from a first plurality of devices;
- determine that second weights of the second deep learning model are outliers from the first distribution and are associated with a second distribution of weights, the second distribution of weights associated with a second plurality of devices;
- aggregate the first weights with weights of the first distribution of weights to determine a first aggregated deep learning model;
- aggregate the second weights with weights of the second distribution of weights to determine a second aggregated deep learning model;
- and
- distribute the first aggregated deep learning model to the first device and the first plurality of devices;
- and
- distribute the second aggregated deep learning model to the second device and the second plurality of devices.
A system to train deep learning models, the system comprising:
- a first device to:
- label input data received at the first device to generate training data;
- train a first deep learning model;
- and
- transmit the first deep learning model to an external device;
- and
- update the first deep learning model based on a first aggregated deep learning model received from the external device;
- and
- a server to:
- obtain the first deep learning model from the first device;
- obtain a second deep learning model from a second device;
- determine that first weights of the first deep learning model are associated with a first distribution of weights from a first plurality of devices;
- determine that second weights of the second deep learning model are outliers from the first distribution and are associated with a second distribution of weights, the second distribution of weights associated with a second plurality of devices;
- aggregate the first weights with weights of the first distribution of weights to determine the first aggregated deep learning model;
- aggregate the second weights with weights of the second distribution of weights to determine a second aggregated deep learning model;
- and
- distribute the first aggregated deep learning model to the first device and the first plurality of devices;
- and
- distribute the second aggregated deep learning model to the second device and the second plurality of devices.
Dependencies are inferred from the claim text.