# US 2019/0244135 A1 · Computer system for distributed machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244135 A1
- Document ID: US20190244135A1
- Status: Published application
- Kind: A1
- Application number: 16/387,247
- Filed: 2019-04-17
- Published: 2019-08-08
- Earliest claimed: 2017-03-09
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Zach MELAMED; Zuguang WU; Roman TALYANSKY; Natan PETERFREUND; Uri VERNER
- Web page: https://patentgrep.com/patent/US20190244135A1

## CPC

- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer system for distributed training of a machine learning model comprising a BSP system, at least one machine learning module, and a shared memory module. The BSP system includes a central BSP control module and at least one local BSP module. The central BSP control module is configured to instruct the at least one local BSP module to store, in its associated shared memory module, a local model. The at least one machine learning module is configured to read, from its associated shared memory module, the local model, compute a gradient based on the local model, and aggregate the gradient immediately after its computation into an aggregated gradient in its associated shared memory module. The central BSP control module is further configured to instruct the at least one local BSP module to periodically read out its associated shared memory module.

## Claims

### Claim 1

A computer system, comprising:
a computer program product storing program code;
computer hardware configured to execute the program code to cause the computer system to perform distributed training of a machine learning model by implementing a plurality of software modules comprising: a central (bulk synchronous parallel) BSP control module, at least one local BSP module, and at least one machine learning module, wherein each machine learning module is associated with exactly one local BSP module; and
at least one shared memory, wherein each shared memory corresponds to exactly one pair of a local BSP module and a machine learning module;
wherein the central BSP control module is configured to instruct the at least one local BSP module to store, in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, a local model;
wherein the at least one machine learning module is configured to:
read, from the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, the local model,
compute a gradient based on the local model, and
aggregate the gradient based on the local model into an aggregated gradient stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module; and
wherein the central BSP control module is further configured to instruct the at least one local BSP module to periodically read out the aggregated gradient stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module.

### Claim 2 (depends on 1)

The computer system according to claim 1, wherein the at least one machine learning module is further configured to:
compute a plurality of gradients based on the local model; and
aggregate the plurality of gradients into the aggregated gradient stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module.

### Claim 3 (depends on 1)

The computer system according to claim 1, wherein the at least one machine learning module is further configured to:
obtain training data from the central BSP control module; and
compute the gradient based on the local model and the training data.

### Claim 4 (depends on 1)

The computer system according to claim 1, wherein the at least one machine learning module is further configured to:
obtain training data pushed from the shared memory corresponding to the at least one local BSP module and the at least one machine learning module; and
compute the gradient based on the local model and the training data.

### Claim 5 (depends on 1)

The computer system according to claim 1, wherein the at least one local BSP module is further configured to:
communicate with a parameter server (PS) in order to receive a PS model that is stored as the local model.

### Claim 6 (depends on 1)

The computer system according to claim 1, wherein, after periodically reading out the aggregated gradient stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, the central BSP control module is further configured to:
instruct the at least one local BSP module to provide, to a parameter server (PS), the aggregated gradient for updating, in the PS, a PS model.

### Claim 7 (depends on 1)

The computer system according to claim 1, wherein:
the central BSP control module is further configured to notify the at least one local BSP module on availability of an updated parameter server (PS) model stored in a PS; and
the at least one local BSP module is configured to download the updated PS model from the PS and to use the updated PS to update the local model stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module.

### Claim 8 (depends on 1)

The computer system according to claim 1, wherein:
the at least one machine learning module is further configured to, in conjunction with storing in its associated shared memory the aggregated gradient, set a gradient available flag; and
the central BSP control module is further configured to, in conjunction with periodically instructing the at least one local BSP module to read out the aggregated gradient stored in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, instruct the at least one local BSP module to read out the aggregated gradient in response to determining that the gradient available flag is set.

### Claim 9 (depends on 1)

The computer system according to claim 1, wherein:
the central BSP control module is further configured to instruct the at least one local BSP module, in conjunction with storing or updating the local model in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, to set a model available flag; and
the at least one machine learning module is further configured to read, from the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, the local model in response to determining that the model available flag is set.

### Claim 10 (depends on 1)

The computer system according to claim 1, wherein:
the central BSP control module is further configured to instruct the at least one local BSP module to store, in the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, a global minimum clock calculated based on clock information obtained from each of the at least one machine learning modules; and
the at least one machine learning module is further configured to read, from the shared memory corresponding to the at least one local BSP module and the at least one machine learning module, the global minimum clock, and interrupt, if a difference of a local clock of the at least one machine learning module and the global minimum clock exceeds a predefined threshold, its computation until the global minimum clock advances and a difference of the local clock of the at least one machine learning module and the global minimum clock is bounded by the predefined threshold.

### Claim 11

A method for distributed training of a machine learning model by implementing a plurality of software modules executed by computer hardware, the modules comprising a central (bulk synchronous parallel) BSP control module, a local BSP module, and a machine learning module, the method comprising:
instructing, by the central BSP control module, the local BSP module to store a local model in a shared memory corresponding to the local BSP module and the machine learning module;
reading, by the machine learning module, the local model from the shared memory corresponding to the local BSP module and the machine learning module;
computing, by the machine learning module, a gradient based on the local model;
aggregating, by the machine learning module, the gradient into an aggregated gradient stored in the shared memory; and
instructing, by the central BSP module, the local BSP module to periodically read out the aggregated gradient from the shared memory.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the machine learning module is further configured to:
compute a plurality of gradients based on the local model; and
aggregate the plurality of gradients into the aggregated gradient stored in the shared memory corresponding to the local BSP module and the machine learning module.

### Claim 13 (depends on 11)

The method according to claim 11, wherein the machine learning module is further configured to:
obtain training data from the central BSP control module; and
compute the gradient based on the local model and the training data.

### Claim 14 (depends on 11)

The method according to claim 11, wherein the machine learning module is further configured to:
obtain training data pushed from the shared memory corresponding to the local BSP module and the machine learning module; and
compute the gradient based on the local model and the training data.

### Claim 15 (depends on 11)

The method according to claim 11, wherein the local BSP module is further configured to:
communicate with a parameter server (PS) in order to receive a PS model that is stored as the local model.

### Claim 16 (depends on 11)

The method according to claim 11, wherein, after periodically reading out the aggregated gradient stored in the shared memory corresponding to the local BSP module and the machine learning module, the central BSP control module is further configured to:
instruct the local BSP module to provide, to a parameter server (PS), the aggregated gradient for updating, in the PS, a PS model.

### Claim 17 (depends on 11)

The method according to claim 11, wherein:
the central BSP control module is further configured to notify the local BSP module on availability of an updated parameter server (PS) model stored in a PS; and
the local BSP module is configured to download the updated PS model from the PS and to use the updated PS to update the local model stored in the shared memory corresponding to the local BSP module and the machine learning module.

### Claim 18 (depends on 11)

The method according to claim 11, wherein:
the machine learning module is further configured to, in conjunction with storing in its associated shared memory the aggregated gradient, set a gradient available flag; and
the central BSP control module is further configured to, in conjunction with periodically instructing the local BSP module to read out the aggregated gradient stored in the shared memory corresponding to the local BSP module and the machine learning module, instruct the local BSP module to read out the aggregated gradient in response to determining that the gradient available flag is set.

### Claim 19 (depends on 11)

The method according to claim 11, wherein:
the central BSP control module is further configured to instruct the local BSP module, in conjunction with storing or updating the local model in the shared memory corresponding to the local BSP module and the machine learning module, to set a model available flag; and
the machine learning module is further configured to read, from the shared memory corresponding to the local BSP module and the machine learning module, the local model in response to determining that the model available flag is set.

### Claim 20

A non-transitory computer program product storing program code for performing, when running on a computer, a method for distributed training of a machine learning model by implementing a plurality of software modules executed by computer hardware, the modules comprising a central (bulk synchronous parallel) BSP control module, a local BSP module, and a machine learning module, the method comprising:
instructing, by the central BSP control module, the local BSP module to store a local model in a shared memory corresponding to the local BSP module and the machine learning module;
reading, by the machine learning module, the local model from the shared memory corresponding to the local BSP module and the machine learning module;
computing, by the machine learning module, a gradient based on the local model;
aggregating, by the machine learning module, the gradient into an aggregated gradient stored in the shared memory; and
instructing, by the central BSP module, the local BSP module to periodically read out the aggregated gradient from the shared memory.
