Back to results
US 10,474,951 B2Granted
Memory efficient scalable deep learning with model parallelization
AssigneeNEC Corporation
InventorsRenqiang Min, Huahua Wang, Asim Kadav
Life of this patent
Estimating the term…
Abstract
Methods and systems for training a neural network include sampling multiple local sub-networks from a global neural network. The local sub-networks include a subset of neurons from each layer of the global neural network. The plurality of local sub-networks are trained at respective local processing devices to produce trained local parameters. The trained local parameters from each local sub-network are averaged to produce trained global parameters.
Claims13
A method for training a neural network, comprising:
- sampling a plurality of local sub-networks from a global neural network, wherein the local sub-networks comprise a subset of neurons from each layer of the global neural network;
- training the plurality of local sub-networks at respective local processing devices to produce trained local parameters;
- and
- averaging the trained local parameters from each local sub-network to produce trained global parameters.
1 more element
A system for training a neural network, comprising:
- a plurality of local sub-network processing devices, each comprising:
- a neural network module comprising neurons that represent a subset of neurons from each layer of a global neural network, the neural network module being configured to train a local sub-network to produce trained local parameters;
- and
- a global parameter server configured to average the trained local parameters from each local sub-network to produce trained global parameters.
1 more element
Dependencies are inferred from the claim text.