# US 11,907,821 B2 · Population-based training of machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,821 B2
- Document ID: US11907821B2
- Status: Granted
- Kind: B2
- Application number: 16/586,236
- Filed: 2019-09-27
- Published: 2021-04-01
- Granted: 2024-02-20
- Earliest claimed: 2019-09-27
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Ang Li; Valentin Clement Dalibard; David Budden; Ola Spyra; Maxwell Elliot Jaderberg; Timothy James Alexander Harley; Sagi Perel; Chenjie Gu; Pramod Gupta
- Web page: https://patentgrep.com/patent/US11907821B2

## CPC

- G06N 20/20: Ensemble learning
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/04: Inference or reasoning models
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. A method includes: maintaining a plurality of training sessions; assigning, to each worker of one or more workers, a respective training session of the plurality of training sessions; repeatedly performing operations until meeting one or more termination criteria, the operations comprising: receiving an updated training session from a respective worker of the one or more workers, selecting a second training session, selecting, based on comparing the updated training session and the second training session using a fitness evaluation function, either the updated training session or the second training session as a parent training session, generating a child training session from the selected parent training session, and assigning the child training session to an available worker, and selecting a candidate model to be a trained model for the machine learning model.

## Claims

### Claim 1

A method of training a machine learning model, the method comprising:
maintaining a plurality of training sessions for training a machine learning model, wherein a training session of the plurality of training sessions comprises data defining (i) respective hyperparameter values for training the machine learning model and (ii) model parameter values of the machine learning model, and defines a session during which the machine learning model is trained by repeatedly updating the model parameter values of the machine learning model according to the respective hyperparameter values;
assigning, to each worker of one or more workers, one or more respective training sessions of the plurality of training sessions, each worker comprising one or more processing units to train the machine learning model during the assigned one or more respective training sessions,
wherein for each worker of the one or more workers and each of the one or more respective training sessions assigned to the worker, the worker is configured to receive the training session, initialize a respective candidate model with model parameter values defined by the training session, train the respective candidate model starting from the model parameter values of the training session and according to hyperparameter values defined by the training session to determine final model parameter values for the training session, and update the training session by replacing the model parameter values defined by the training session with the final model parameter values determined for the training session;
repeatedly performing operations until meeting one or more termination criteria, the operations comprising:
receiving an updated training session from a respective worker of the one or more workers, wherein the updated training session identifies a first training session assigned to the respective worker and the final model parameter values determined by the respective worker for the first training session assigned to the respective worker,
selecting a second training session from the plurality of training sessions,
selecting, based on comparing the updated training session and the second training session using a fitness evaluation function, either the updated training session or the second training session as a parent training session,
generating a child training session from the selected parent training session, wherein the generating comprises:
generating child hyperparameter values of the child training session for training the machine learning model by updating hyperparameter values that have been used to train the machine learning model in the parent training session, and
generating child model parameters for the child training session by setting the child model parameters using the model parameter values of the parent training session, and
assigning the child training session to an available worker; and
selecting a candidate model trained by a worker of the one or more workers to be a trained model for the machine learning model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the operations further comprise:
determining that the one or more termination criteria have been met; and
in response to determining that the one or more termination criteria have been met, selecting the candidate model trained by the worker of the one or more workers to be the trained model for the machine learning model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more termination criteria comprise each training session in the plurality of training sessions having been compared to a respective other training session in the plurality of training sessions.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving a request for final hyperparameter values used to train the selected candidate model, and
in response to the request, providing hyperparameter values of a training session used by the worker that trained the selected candidate model.

### Claim 5 (depends on 1)

The method of claim 1, wherein the operations further comprise:
maintaining a training session dependency graph, wherein each node of the training session dependency graph corresponds to a respective training session of the plurality of training sessions, and each edge between a pair of nodes in the training session dependency graph corresponds to a training session represented by a first node of the pair of nodes having been generated from another training session represented by a second node of the pair of nodes.

### Claim 6 (depends on 5)

The method of claim 5, wherein the plurality of training sessions comprises one or more initial training sessions, wherein the one or more initial training sessions comprise respective hyperparameter values sampled from a first distribution of values, and respective model parameter values sampled from a second distribution of values, and wherein the operations further comprise:
generating a hyperparameter schedule for the selected candidate model, comprising:
identifying a particular training session assigned to a worker of the one or more workers, wherein the particular training session comprises model parameter values corresponding to model parameter values of the selected candidate model,
generating a subgraph comprising a node representing the particular training session and respective nodes whose respective edges form a path from the node comprising the node representing the particular training session and a node representing an initial training session of the one or more initial training sessions, and
obtaining a hyperparameter schedule from respective hyperparameter values of each training session represented by a respective node in the subgraph.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the child training session from the selected parent training session comprises:
generating modified hyperparameter values for the child training session by multiplying respective hyperparameter values of the selected parent training session with a randomly sampled value.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more processing units are heterogeneous.

### Claim 9 (depends on 1)

The method of claim 1, wherein respective final parameter values for each training session request of a plurality of training session requests is represented as a checkpoint software object.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
providing the trained model for use in processing new inputs.

### Claim 11

A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
maintaining a plurality of training sessions for training a machine learning model, wherein a training session of the plurality of training sessions comprises data defining (i) respective hyperparameter values for training the machine learning model and (ii) model parameter values of the machine learning model, and defines a session during which the machine learning model is trained by repeatedly updating the model parameter values of the machine learning model according to the respective hyperparameter values;
assigning, to each worker of one or more workers, one or more respective training sessions of the plurality of training sessions, each worker comprising one or more processing units to train the machine learning model during the assigned one or more respective training sessions,
wherein for each worker of the one or more workers and each of the one or more respective training sessions assigned to the worker, the worker is configured to receive the training session, initialize a respective candidate model with model parameter values defined by the training session, train the respective candidate model starting from the model parameter values of the training session and according to hyperparameter values defined by the training session to determine final model parameter values for the training session, and update the training session by replacing the model parameter values defined by the training session with the final model parameter values determined for the training session;
repeatedly performing operations until meeting one or more termination criteria, the operations comprising:
receiving an updated training session from a respective worker of the one or more workers, wherein the updated training session identifies a first training session assigned to the respective worker and the final model parameter values determined by the respective worker for the first training session assigned to the respective worker,
selecting a second training session from the plurality of training sessions,
selecting, based on comparing the updated training session and the second training session using a fitness evaluation function, either the updated training session or the second training session as a parent training session,
generating a child training session from the selected parent training session, wherein the generating comprises:
generating child hyperparameter values of the child training session for training the machine learning model by updating hyperparameter values that have been used to train the machine learning model in the parent training session, and
generating child model parameters for the child training session by setting the child model parameters using the model parameter values of the parent training session, and
assigning the child training session to an available worker; and
selecting a candidate model trained by a worker of the one or more workers to be a trained model for the machine learning model.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise:
determining that the one or more termination criteria have been met; and
in response to determining that the one or more termination criteria have been met, selecting the candidate model trained by the worker of the one or more workers to be the trained model for the machine learning model.

### Claim 13 (depends on 11)

The system of claim 11, wherein the one or more termination criteria comprise each training session in the plurality of training sessions having been compared to a respective other training session in the plurality of training sessions.

### Claim 14 (depends on 11)

The system of claim 11, wherein the operations further comprise:
receiving a request for final hyperparameter values used to train the selected candidate model, and
in response to the request, providing hyperparameter values of a training session used by the worker that trained the selected candidate model.

### Claim 15 (depends on 11)

The system of claim 11, wherein the operations further comprise:
maintaining a training session dependency graph, wherein each node of the training session dependency graph corresponds to a respective training session of the plurality of training sessions, and each edge between a pair of nodes in the training session dependency graph corresponds to a training session represented by a first node of the pair of nodes having been generated from another training session represented by a second node of the pair of nodes.

### Claim 16 (depends on 15)

The system of claim 15, wherein the plurality of training sessions comprises one or more initial training sessions, wherein the one or more initial training sessions comprise respective hyperparameter values sampled from a first distribution of values, and respective model parameter values sampled from a second distribution of values, and wherein the operations further comprise:
generating a hyperparameter schedule for the selected candidate model, comprising:
identifying a particular training session assigned to a worker of the one or more workers, wherein the particular training session comprises model parameter values corresponding to model parameter values of the selected candidate model,
generating a subgraph comprising a node representing the particular training session and respective nodes whose respective edges form a path from the node comprising the node representing the particular training session and a node representing an initial training session of the one or more initial training sessions, and
obtaining a hyperparameter schedule from respective hyperparameter values of each training session represented by a respective node in the subgraph.

### Claim 17 (depends on 11)

The system of claim 11, wherein generating the child training session from the selected parent training session comprises:
generating modified hyperparameter values for the child training session by multiplying respective hyperparameter values of the selected parent training session with a randomly sampled value.

### Claim 18 (depends on 11)

The system of claim 11, wherein the one or more processing units are heterogeneous.

### Claim 19 (depends on 11)

The system of claim 11, wherein respective final parameter values for each training session request of a plurality of training session requests is represented as a checkpoint software object.

### Claim 20 (depends on 11)

The system of claim 11, wherein the operations further comprise:
providing the trained model for use in processing new inputs.

### Claim 21

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
maintaining a plurality of training sessions for training a machine learning model, wherein a training session of the plurality of training sessions comprises data defining (i) respective hyperparameter values for training the machine learning model and (ii) model parameter values of the machine learning model, and defines a session during which the machine learning model is trained by repeatedly updating the model parameter values of the machine learning model according to the respective hyperparameter values;
assigning, to each worker of one or more workers, one or more respective training sessions of the plurality of training sessions, each worker comprising one or more processing units to train the machine learning model during the assigned one or more respective training sessions,
wherein for each worker of the one or more workers and each of the one or more respective training sessions assigned to the worker, the worker is configured to receive the training session, initialize a respective candidate model with model parameter values defined by the training session, train the respective candidate model starting from the model parameter values of the training session and according to hyperparameter values defined by the training session to determine final model parameter values for the training session, and update the training session by replacing the model parameter values defined by the training session with the final model parameter values determined for the training session;
repeatedly performing operations until meeting one or more termination criteria, the operations comprising:
receiving an updated training session from a respective worker of the one or more workers, wherein the updated training session identifies a first training session assigned to the respective worker and the final model parameter values determined by the respective worker for the first training session assigned to the respective worker,
selecting a second training session from the plurality of training sessions,
selecting, based on comparing the updated training session and the second training session using a fitness evaluation function, either the updated training session or the second training session as a parent training session,
generating a child training session from the selected parent training session, wherein the generating comprises:
generating child hyperparameter values of the child training session for training the machine learning model by updating hyperparameter values that have been used to train the machine learning model in the parent training session, and
generating child model parameters for the child training session by setting the child model parameters using the model parameter values of the parent training session, and
assigning the child training session to an available worker; and
selecting a candidate model trained by a worker of the one or more workers to be a trained model for the machine learning model.

### Claim 22 (depends on 21)

The computer-readable storage media of claim 21, wherein the operations further comprise:
determining that the one or more termination criteria have been met; and
in response to determining that the one or more termination criteria have been met, selecting the candidate model trained by the worker of the one or more workers to be the trained model for the machine learning model.

### Claim 23 (depends on 21)

The computer-readable storage media of claim 21, wherein the one or more termination criteria comprise each training session in the plurality of training sessions having been compared to a respective other training session in the plurality of training sessions.

### Claim 24 (depends on 21)

The computer-readable storage media of claim 21, wherein the operations further comprise:
receiving a request for final hyperparameter values used to train the selected candidate model, and
in response to the request, providing hyperparameter values of a training session used by the worker that trained the selected candidate model.

### Claim 25 (depends on 21)

The computer-readable storage media of claim 21, wherein the operations further comprise:
maintaining a training session dependency graph, wherein each node of the training session dependency graph corresponds to a respective training session of the plurality of training sessions, and each edge between a pair of nodes in the training session dependency graph corresponds to a training session represented by a first node of the pair of nodes having been generated from another training session represented by a second node of the pair of nodes.

### Claim 26 (depends on 25)

The computer-readable storage media of claim 25, wherein the plurality of training sessions comprises one or more initial training sessions, wherein the one or more initial training sessions comprise respective hyperparameter values sampled from a first distribution of values, and respective model parameter values sampled from a second distribution of values, and wherein the operations further comprise:
generating a hyperparameter schedule for the selected candidate model, comprising:
identifying a particular training session assigned to a worker of the one or more workers, wherein the particular training session comprises model parameter values corresponding to model parameter values of the selected candidate model,
generating a subgraph comprising a node representing the particular training session and respective nodes whose respective edges form a path from the node comprising the node representing the particular training session and a node representing an initial training session of the one or more initial training sessions, and
obtaining a hyperparameter schedule from respective hyperparameter values of each training session represented by a respective node in the subgraph.

### Claim 27 (depends on 21)

The computer-readable storage media of claim 21, wherein generating the child training session from the selected parent training session comprises:
generating modified hyperparameter values for the child training session by multiplying respective hyperparameter values of the selected parent training session with a randomly sampled value.

### Claim 28 (depends on 21)

The computer-readable storage media of claim 21, wherein the one or more processing units are heterogeneous.

### Claim 29 (depends on 21)

The computer-readable storage media of claim 21, wherein respective final parameter values for each training session request of a plurality of training session requests is represented as a checkpoint software object.

### Claim 30 (depends on 21)

The computer-readable storage media of claim 21, wherein the operations further comprise:
providing the trained model for use in processing new inputs.