# US 12,182,712 B2 · Node-splitting for neural networks based on magnitude of norm of vector of partial derivatives > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,712 B2 - Document ID: US12182712B2 - Status: Granted - Kind: B2 - Application number: 18/352,044 - Filed: 2023-07-13 - Published: 2023-11-16 - Granted: 2024-12-31 - Earliest claimed: 2018-01-30 - Assignees: D5Ai LLC - Applicants: D5Ai LLC - Inventors: James K. Baker; Bradley J. Baker - Web page: https://patentgrep.com/patent/US12182712B2 ## CPC - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/24: Classification techniques - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 3/098: Distributed learning, e.g. federated learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - H04L 67/142: Managing session states for stateless protocols; Signalling session states; State transitions; Keeping-state mechanisms - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 5/046: Forward inferencing; Production systems - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network. ## Claims ### Claim 1 A method of expediting training, performed by computer processors, of a neural network, wherein the neural network comprises a plurality of nodes, the method comprising: following a first number of training iterations of the neural network performed by a computer system that comprises on or more processor cores, detecting, by the computer system, a slow learning condition in the training of the neural network, wherein the slow learning condition comprises a condition where a magnitude of a gradient for the neural network computed by the one or more processor cores is less than a threshold value for a threshold number of the first number of training iterations; and upon detecting the slow learning condition, accelerating the training of the neural network by: computing, by the computer system, for each of a first set of the plurality of nodes, a magnitude of a norm of a vector of partial derivatives, wherein the first set comprises one or more nodes of the plurality of nodes of the neural network; identifying, by the computer, a first node in the first set for which the magnitude of the norm of the vector of partial derivatives is greater than a threshold value; adding, by the computer system, first and second supplemental nodes to the neural network; and after adding the first and second supplemental nodes, resuming, by the computer system, iterative training of the neural network, such that the first supplemental node is trained differently by the computer system from the second supplemental node and the first node in the resumed iterative training by the computer system and such that the second supplemental node is trained differently by computer system from the first supplemental node and the first node in the resumed iterative training by computer system. ### Claim 2 (depends on 1) The method of claim 1, wherein adding the first and second supplemental nodes comprises initializing the first and second supplemental nodes to the first node. ### Claim 3 (depends on 2) The method of claim 2, wherein initializing the first and second supplemental nodes comprises, prior to the resumed iterative training, initializing input weights of the first and second supplemental nodes to be the same as input weights of the first node. ### Claim 4 (depends on 3) The method of claim 3, wherein initializing the first and second supplemental nodes further comprises, prior to the resumed iterative training, initializing output weights of the first and second supplemental nodes to be zero. ### Claim 5 (depends on 2) The method of claim 2, wherein the first set of the plurality of nodes comprises all of the nodes of the neural network. ### Claim 6 (depends on 2) The method of claim 2, wherein the magnitude of the norm of the vector of partial derivatives comprises, for each of the first set of the plurality of nodes, a magnitude of the norm of the partial derivative of an objective function for the neural network with respect to an activation value for the node over a representative set of training data examples. ### Claim 7 (depends on 2) The method of claim 2, where the norm comprises an L1 norm. ### Claim 8 (depends on 1) The method of claim 1, wherein resuming the iterative training comprises: a special training period that comprises: separating training data for the resumed iterative training into at least first and second training data subsets; training, in the special training period, the neural network with the first and second training data subsets, wherein the training in the special training period comprises: back-propagating information to the first supplemental node only for the first training data subset; and back-propagation information to the second supplemental node only for the second training data subset; and following the special training period, resuming training of the neural network. ### Claim 9 (depends on 8) The method of claim 8, wherein separating the training data for the special training period comprises separating the training data into at least the first and second training data subsets based on a sign of a derivative of an objective function of the neural network with respect to activations of the first and second supplemental nodes. ### Claim 10 (depends on 8) The method of claim 8, wherein the special training period further comprises freezing a remainder of the neural network besides the first and second supplemental nodes, such that the remainder of the neural network is not frozen for the resumed training. ### Claim 11 (depends on 8) The method of claim 8, wherein the training in the special training period comprises not back-propagating information beyond input arcs to the first and second supplemental nodes. ### Claim 12 (depends on 1) The method of claim 1, wherein the neural network comprises a feedforward neural network. ### Claim 13 (depends on 1) The method of claim 1, wherein: the one or more processor cores comprises two or more GPU cores; and resuming iterative training of the neural network comprises processing training data by the two or more GPU cores in parallel. ### Claim 14 A computer system that expedites training, performed by a computer system, of a neural network, wherein the neural network comprises a plurality of nodes, the computer system comprising: one or more processor cores; and computer memory in communication with the one or more processor cores, wherein the computer memory stores computer instructions that when executed by the one or more processor cores, cause the one or more processor cores to: following a first number of training iterations performed by the computer system, detect a slow learning condition in training of the neural network, wherein the slow learning condition comprises a condition where a magnitude of a gradient for the neural network computed by the one or more processor cores is less than a threshold value for a threshold number of the first number of training iterations; and upon detecting the slow learning condition, accelerating the training of the neural network by: compute, for each of a first set of the plurality of nodes, a magnitude of a norm of a vector of partial derivatives, wherein the first set comprises one or more nodes of the plurality of nodes of the neural network; identify a first node in the first set for which the magnitude of the norm of the vector of partial derivatives is greater than a threshold value; add first and second supplemental nodes to the neural network; and after adding the first and second supplemental nodes, resume iterative training of the neural network by the computer system, such that the first supplemental node is trained differently by the computer system from the second supplemental node and the first node in the resumed iterative training and such that the second supplemental node is trained differently by the computer system from the first supplemental node and the first node in the resumed iterative training. ### Claim 15 (depends on 14) The computer system of claim 14, wherein the computer memory stores computer instructions that when executed by the one or more processor cores cause the one or more processor to, as part of adding the first and second supplemental nodes, initialize the first and second supplemental nodes to the first node. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the computer memory stores computer instructions that when executed by the one or more processor cores cause the one or more processor to initialize the first and second supplemental nodes by, prior to the resumed iterative training, initializing input weights of the first and second supplemental nodes to be the same as input weights of the first node. ### Claim 17 (depends on 16) The computer system of claim 16, wherein the computer memory stores computer instructions that when executed by the one or more processor cores cause the one or more processor to initialize the first and second supplemental nodes by, prior to the resumed iterative training, initializing output weights of the first and second supplemental nodes to be zero. ### Claim 18 (depends on 15) The computer system of claim 15, wherein the first set of the plurality of nodes comprises all of the nodes of the neural network. ### Claim 19 (depends on 15) The computer system of claim 15, wherein the magnitude of the norm of the vector of partial derivatives comprises, for each of the first set of the plurality of nodes, a magnitude of the norm of the partial derivative of an objective function for the neural network with respect to an activation value for the node over a representative set of training data examples. ### Claim 20 (depends on 15) The computer system of claim 15, where the norm comprises an L1 norm. ### Claim 21 (depends on 16) The computer system of claim 16, wherein the resumed iterative training comprises: a special training period wherein the one or more processor cores are configured to: separate training data for the resumed iterative training into at least first and second training data subsets; train, in the special training period, the neural network with the first and second training data subsets, wherein the training in the special training period comprises: back-propagating information to the first supplemental node only for the first training data subset; and back-propagation information to the second supplemental node only for the second training data subset; and following the special training period, resumed training of the neural network. ### Claim 22 (depends on 21) The computer system of claim 21, wherein the computer memory stores computer instructions that when executed by the one or more processor cores cause the one or more processor to separate the training data for the special training period by separating the training data into at least the first and second training data subsets based on a sign of a derivative of an objective function of the neural network with respect to activations of the first and second supplemental nodes. ### Claim 23 (depends on 21) The computer system of claim 21, wherein, as part of the special training period, the one or more processor cores are further configured to freeze a remainder of the neural network besides the first and second supplemental nodes, such that the remainder of the neural network is not frozen for the resumed training. ### Claim 24 (depends on 21) The computer system of claim 21, wherein the one or more processor cores are further configured to, as part of special training period, not back-propagate information beyond input arcs to the first and second supplemental nodes. ### Claim 25 (depends on 14) The computer system of claim 14, wherein the neural network comprises a feedforward neural network. ### Claim 26 (depends on 14) The computer system of claim 14, wherein the one or more processor cores comprises two or more GPU cores for processing training data in parallel in the resumed iterative training of the neural network.