# US 2021/0342747 A1 · Method and system for distributed deep machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342747 A1
- Document ID: US20210342747A1
- Status: Published application
- Kind: A1
- Application number: 17/376,962
- Filed: 2021-07-15
- Published: 2021-11-04
- Earliest claimed: 2016-01-29
- Assignees: Verizon Media Inc.
- Applicants: Verizon Media Inc.
- Inventors: Andrew Feng; Jun Shi; Mridul Jain; Peter Cnudde
- Web page: https://patentgrep.com/patent/US20210342747A1

## CPC

- G06N 20/00: Machine learning
- H04L 41/16: using machine learning or artificial intelligence
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The present teaching relates to distributed deep machine learning on a cluster. In one example, a request is received for estimating one or more parameters associated with a machine learning model on a cluster including a plurality of nodes. A set of data is obtained to be used for estimating the one or more parameters. The set of data is divided into a plurality of sub-sets of data, each of which corresponds to one of the plurality of nodes. Each sub-set of data is allocated to a corresponding node for estimating values of the one or more parameters based on the sub-set of data. Estimated values of the one or more parameters obtained based on a corresponding sub-set of data allocated to the node, are received from each of the plurality of nodes. The one or more parameters of the machine learning model are estimated based on the estimated values of the one or more parameters generated by at least some of the plurality of nodes.

## Claims

### Claim 1

A method implemented on at least one computing device each of which has at least one processor, storage, and a communication platform connected to a network for estimating one or more parameters of a model, the method comprising:
estimating, by a plurality of processing units in a first node of a plurality of nodes, values of the one or more parameters associated with the model;
generating, by a preselected representative processing unit of the first node, an estimated vector based on the values of the one or more parameters estimated by the plurality of processing units in the first node;
dividing, by the representative processing unit of the first node, the estimated vector into a plurality of portions, each portion of the plurality of portions being designated to one of the plurality of nodes;
sending, by the representative processing unit of the first node, a first estimate of a portion of the vector designated to the first node to representative processing units of the other nodes of the plurality of nodes, wherein the first estimate is determined based on estimates of the portion of the vector designated to the first node that are collected from representative processing units of other nodes of the plurality of nodes;
receiving, by the representative processing unit of the first node, from the representative processing units of the other nodes of the plurality of nodes, second estimates of corresponding portions of the vector designated to the other nodes; and
sending, by the representative processing unit of the first node, the first estimate and the received second estimates to other processing units of the plurality of processing units included in the first node.

### Claim 2 (depends on 1)

The method of claim 1, wherein the generating the estimated vector comprises:
aggregating the values of the one or more parameters estimated by the plurality of processing units in the first node to generate the estimated vector.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the first node, training data to be used to estimate the one or more parameters;
generating updated parameter estimates based on a corresponding sub-set of data;
exchanging the updated parameter estimates with other nodes of the plurality of nodes to generate a state of the one or more parameters that is shared by the plurality of nodes; and
repeating the steps of generating and exchanging until a predetermined condition is met to generate the estimated values of the one or more parameters.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating a snapshot of a state of the one or more parameters of the model; and
storing the snapshot in a storage outside the plurality of nodes.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
in response to a failure with respect to estimating the one or more parameters, retrieving the snapshot of the state of the one or more parameters of the model from the storage; and
continuing to estimate the one or more parameters based on the last state of the one or more parameters.

### Claim 6 (depends on 1)

The method claim 1, wherein the plurality of nodes are synchronized via a Message Passing Interface (MPI) AllReduce based Application Program Interface (API) using a network interface implemented on each of the plurality of nodes, wherein the network interface includes an Ethernet interface, an Infiniband interface, or the Ethernet interface and the Infiniband interface.

### Claim 7 (depends on 1)

The method of claim 1, wherein the preselected representative processing unit is a graphical processing unit.

### Claim 8

A system, having at least one processor, storage, and a communication platform connected to a network for estimating one or more parameters of a model, the system comprising:
a first node, of a plurality of nodes, implemented by the at least one processor and configured for:
estimating, by a plurality of processing units in the first node, values of the one or more parameters associated with the model;
generating, by a preselected representative processing unit of the first node, an estimated vector based on the values of the one or more parameters estimated by the plurality of processing units in the first node;
dividing, by the representative processing unit of the first node, the estimated vector into a plurality of portions, each portion of the plurality of portions being designated to one of the plurality of nodes;
sending, by the representative processing unit of the first node, a first estimate of a portion of the vector designated to the first node to representative processing units of the other nodes of the plurality of nodes, wherein the first estimate is determined based on estimates of the portion of the vector designated to the first node that are collected from representative processing units of other nodes of the plurality of nodes;
receiving, by the representative processing unit of the first node, from the representative processing units of the other nodes of the plurality of nodes, second estimates of corresponding portions of the vector designated to the other nodes; and
sending, by the representative processing unit of the first node, the first estimate and the received second estimates to other processing units of the plurality of processing units included in the first node; and
the other nodes, of the plurality of nodes, implemented by the at least one processor and configured for:
receiving the first estimate of the portion of the vector designated to the first node; and
sending the second estimates of corresponding portions of the vector designated to the other nodes.

### Claim 9 (depends on 8)

The system of claim 8, wherein the generating the estimated vector comprises:
aggregating the values of the one or more parameters estimated by the plurality of processing units in the first node to generate the estimated vector.

### Claim 10 (depends on 8)

The system of claim 8, wherein the first node is implemented by the at least one processor and configured further for:
obtaining training data to be used to estimate the one or more parameters;
generating updated parameter estimates based on a corresponding sub-set of data;
exchanging the updated parameter estimates with the other nodes to generate a state of the one or more parameters that is shared by the plurality of nodes; and
repeating the steps of generating and exchanging until a predetermined condition is met to generate the estimated values of the one or more parameters.

### Claim 11 (depends on 8)

The system of claim 8, wherein the first node comprises:
a model snapshot generator implemented by the at least one processor and configured for generating a snapshot of a state of the one or more parameters of the model; and
storing the snapshot in a storage outside the plurality of nodes.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first node is implemented by the at least one processor and configured further for:
in response to a failure with respect to estimating the one or more parameters, retrieving the snapshot of the state of the one or more parameters of the model from the storage; and
continuing to estimate the one or more parameters based on the last state of the one or more parameters.

### Claim 13 (depends on 8)

The system claim 8, wherein the plurality of nodes are synchronized via a Message Passing Interface (MPI) AllReduce based Application Program Interface (API) using a network interface implemented on each of the plurality of nodes, wherein the network interface includes an Ethernet interface, an Infiniband interface, or the Ethernet interface and the Infiniband interface.

### Claim 14 (depends on 8)

The system of claim 8, wherein the preselected representative processing unit is a graphical processing unit.

### Claim 15

A machine-readable tangible and non-transitory medium having information for estimating one or more parameters of a model, wherein the information, when read by a first node of a plurality of nodes, causes the first node to perform the following:
estimating, by a plurality of processing units in the first node of the plurality of nodes, values of the one or more parameters associated with the model;
generating, by a preselected representative processing unit of the first node, an estimated vector based on the values of the one or more parameters estimated by the plurality of processing units in the first node;
dividing, by the representative processing unit of the first node, the estimated vector into a plurality of portions, each portion of the plurality of portions being designated to one of the plurality of nodes;
sending, by the representative processing unit of the first node, a first estimate of a portion of the vector designated to the first node to representative processing units of the other nodes of the plurality of nodes, wherein the first estimate is determined based on estimates of the portion of the vector designated to the first node that are collected from representative processing units of other nodes of the plurality of nodes;
receiving, by the representative processing unit of the first node, from the representative processing units of the other nodes of the plurality of nodes, second estimates of corresponding portions of the vector designated to the other nodes; and
sending, by the representative processing unit of the first node, the first estimate and the received second estimates to other processing units of the plurality of processing units included in the first node.

### Claim 16 (depends on 15)

The machine-readable tangible and non-transitory medium of claim 15, wherein the information, when read by the first node, causes the first node to perform generating the estimated vector by:
aggregating the values of the one or more parameters estimated by the plurality of processing units in the first node to generate the estimated vector.

### Claim 17 (depends on 15)

The machine-readable tangible and non-transitory medium of claim 15, wherein the information, when read by the first node, causes the first node to further perform:
obtaining training data to be used to estimate the one or more parameters;
generating updated parameter estimates based on a corresponding sub-set of data;
exchanging the updated parameter estimates with other nodes of the plurality of nodes to generate a state of the one or more parameters that is shared by the plurality of nodes; and
repeating the steps of generating and exchanging until a predetermined condition is met to generate the estimated values of the one or more parameters.

### Claim 18 (depends on 15)

The machine-readable tangible and non-transitory medium of claim 15, wherein the information, when read by the first node, causes the first node to further perform:
generating a snapshot of a state of the one or more parameters of the model; and
storing the snapshot in a storage outside the plurality of nodes.

### Claim 19 (depends on 18)

The machine-readable tangible and non-transitory medium of claim 18, wherein the information, when read by the first node, causes the first node to further perform:
in response to a failure with respect to estimating the one or more parameters, retrieving the snapshot of the state of the one or more parameters of the model from the storage; and
continuing to estimate the one or more parameters based on the last state of the one or more parameters.

### Claim 20 (depends on 15)

The machine-readable tangible and non-transitory medium of claim 15, wherein the plurality of nodes are synchronized via a Message Passing Interface (MPI) AllReduce based Application Program Interface (API) using a network interface implemented on each of the plurality of nodes, wherein the network interface includes an Ethernet interface, an Infiniband interface, or the Ethernet interface and the Infiniband interface.
