Machine Learning System Enabling Effective Training
AssigneeGraphcore Limited
InventorsCharles BLAKE, Douglas Alexander Harper ORR, Carlo LUSCHI
Life of this patent
Estimating the term…
Abstract
A machine learning system implements a machine learning model. The system includes at least one layer of processing nodes, each processing node comprising a processor that executes computer readable instructions to perform at least one operation based on one or more inputs received at the processing node. The operation is scaled by a first scaling factor which has been calculated to cause a variance of an output of the at least one operation to have a target variance, for example unit variance or a variance that matches the variance of the input.
Claims22
A machine learning system implementing a machine learning model, the system comprising:
- at least one layer of processing nodes, each processing node comprising a processor configured to execute computer readable instructions to perform at least one operation based on one or more inputs received at the processing node,
- wherein the at least one operation is scaled by a first scaling factor which has been calculated to cause a variance of an output of the at least one operation to have a target variance.
A computer-implemented method comprising:
- receiving a computational graph, the computational graph comprising:
- a plurality of nodes, each node of the plurality of nodes corresponding to a computational operation for training a machine learning model, and
- a plurality of edges, each edge connecting a pair of the nodes and corresponding to an output of a first node of the pair of the nodes and an input to a second node of the pair of the nodes;
- inserting a first scaling factor into the computational graph associated with at least one node of the plurality of nodes, the first scaling factor calculated to cause a variance of an output of the at least one node to have a target variance.
A non-transitory computer-readable medium comprising computer-executable instructions, the instructions when executed implementing a neural network,
- wherein the instructions comprise first code embodying at least one scaled operation configured to receive a tensor of weights and a tensor of input activations and to generate a tensor of output activations with a target variance.
Dependencies are inferred from the claim text.