Node-splitting for neural networks based on magnitude of norm of vector of partial derivatives
AssigneeD5Ai LLC
InventorsJames K. Baker, Bradley J. Baker
Life of this patent
Estimating the term…
Abstract
A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.
Claims26
A method of expediting training, performed by computer processors, of a neural network, wherein the neural network comprises a plurality of nodes, the method comprising:
- following a first number of training iterations of the neural network performed by a computer system that comprises on or more processor cores, detecting, by the computer system, a slow learning condition in the training of the neural network, wherein the slow learning condition comprises a condition where a magnitude of a gradient for the neural network computed by the one or more processor cores is less than a threshold value for a threshold number of the first number of training iterations;
- and
- upon detecting the slow learning condition, accelerating the training of the neural network by:
- computing, by the computer system, for each of a first set of the plurality of nodes, a magnitude of a norm of a vector of partial derivatives, wherein the first set comprises one or more nodes of the plurality of nodes of the neural network;
- identifying, by the computer, a first node in the first set for which the magnitude of the norm of the vector of partial derivatives is greater than a threshold value;
- adding, by the computer system, first and second supplemental nodes to the neural network;
- and
- after adding the first and second supplemental nodes, resuming, by the computer system, iterative training of the neural network, such that the first supplemental node is trained differently by the computer system from the second supplemental node and the first node in the resumed iterative training by the computer system and such that the second supplemental node is trained differently by computer system from the first supplemental node and the first node in the resumed iterative training by computer system.
A computer system that expedites training, performed by a computer system, of a neural network, wherein the neural network comprises a plurality of nodes, the computer system comprising:
- one or more processor cores;
- and
- computer memory in communication with the one or more processor cores, wherein the computer memory stores computer instructions that when executed by the one or more processor cores, cause the one or more processor cores to:
- following a first number of training iterations performed by the computer system, detect a slow learning condition in training of the neural network, wherein the slow learning condition comprises a condition where a magnitude of a gradient for the neural network computed by the one or more processor cores is less than a threshold value for a threshold number of the first number of training iterations;
- and
- upon detecting the slow learning condition, accelerating the training of the neural network by:
- compute, for each of a first set of the plurality of nodes, a magnitude of a norm of a vector of partial derivatives, wherein the first set comprises one or more nodes of the plurality of nodes of the neural network;
- identify a first node in the first set for which the magnitude of the norm of the vector of partial derivatives is greater than a threshold value;
- add first and second supplemental nodes to the neural network;
- and
- after adding the first and second supplemental nodes, resume iterative training of the neural network by the computer system, such that the first supplemental node is trained differently by the computer system from the second supplemental node and the first node in the resumed iterative training and such that the second supplemental node is trained differently by the computer system from the first supplemental node and the first node in the resumed iterative training.
Dependencies are inferred from the claim text.