Back to results
US 10,540,608 B1Granted
Dynamically scaled training fleets for machine learning
AssigneeAmazon Technologies, Inc.
InventorsLeo Parker Dirac, Rakesh Madhavan Nambiar, Avinash Aghoram Ravichandran
Life of this patent
Estimating the term…
Abstract
A first set of execution platforms is deployed for a set of operations of a training phase of a machine learning model. Prior to the completion of the training phase, a triggering condition for deployment of a different set of execution platforms is detected. The different set of execution platforms is deployed for a subsequent set of training phase operations.
Claims20
A system, comprising:
- one or more computing devices of a machine learning service implemented at a provider network;
- wherein the one or more computing devices include one or more respective hardware processors and associated memory storing program instructions that are executable on the one or more hardware processors to cause the one or more computing devices to:
- receive an indication of a requirement to train a machine learning model;
- identify a parallel technique to be employed during a training phase of the machine learning model, wherein the parallel technique is configured to be performed in parallel on a set of execution platforms and includes transfer of model parameter synchronization data among the set of execution platforms, and wherein convergence towards an optimization goal of the parallel technique is expected to be correlated with a reduction in the amount of the model parameter synchronization data transferred among the set of execution platforms;
- determine (a) a first subset of a plurality of execution platforms of the machine learning service to be deployed to perform at least a first set of operations of the training phase using the parallel technique;
- and (b) one or more conditions during the training phase which are to trigger a deployment of a second subset of the plurality of execution platforms to perform at least a second set of operations of the training phase using the parallel technique subsequent to the first set of operations, wherein the number of execution platforms included in the second subset differs from the number of execution platforms included in the first subset;
- detect, during the training phrase and using one or more metrics collected during performance of the first set of operations using the parallel technique, that a particular condition of the one or more conditions has been met;
- deploy, during the training phrase and responsive to the detection, the second subset of execution platforms to perform the second set of operations using the parallel technique;
- and
- utilize, after the training phase is terminated in response to a detection that a training goal of the machine learning model has been achieved, the machine learning model to generate one or more predictions.
7 more elements
A method, comprising:
- performing, at one or more computing devices:
- determining (a) a first subset of a plurality of execution platforms to be deployed to perform at least a first set of operations of a training phase of a machine learning model in parallel using a parallel technique;
- and (b) one or more conditions which are to trigger, prior to a completion of the training phase, a deployment of a second subset of the plurality of execution platforms to perform at least a different set of operations of the training phase using the parallel technique;
- detecting, during the training phrase, that a particular condition of the one more conditions has been met during performance of the first set of operations using the parallel technique;
- deploying, during the training phrase and responsive to the detection, the second subset of execution platforms to perform the different set of operations of the training phase using the parallel technique;
- and
- terminating the training phase in response to a detection that a training goal of the machine learning model has been attained.
4 more elements
A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
- determine a first subset of a plurality of execution platforms to be deployed to perform at least a first set of operations of a training phase of a machine learning model using a parallel technique;
- detect, prior to completion of the training phase, that a triggering condition for a deployment of a second subset of the plurality of execution platforms to perform at least a different set of operations of the training phase has been met, wherein the second subset of execution platforms are configured to perform the different set of operations using the parallel technique;
- and
- initiate, during the training phase and responsive to the detection, a deployment of the second subset of execution platforms to perform the different set of operations of the training phase.
1 more element
Dependencies are inferred from the claim text.