Placement-Aware Accelaration of Parameter Optimization in a Predictive Model

A method of optimizing a predictive model of a system using training data and knowledge of the placement of parameters within the predictive model, the method comprising:

• generating a set of slices comprising a predictive model topology wherein:
• the topology comprises a plurality of input nodes, a plurality of output nodes, and a plurality of paths connecting the input nodes to the output nodes;
• each path comprises one or more links associated with respective weights;
• the set of slices comprises a primary slice comprising the topology, a first layer of slices comprising respective subsets of the primary slice, and a second layer of slices comprising respective subsets of the slices comprising the first layer;
• and
• performing first-epoch optimization of the predictive model, comprising:
• dividing the training data into a plurality of batches comprising a first batch set;
• for each slice selected from the second layer, training the selected slice using a first group of batches comprising the first batch set to determine first initial optimized values for each weight associated with a link comprising the selected slice;
• and
• for each slice selected from the first layer, training the selected slice using a second group of batches comprising the first batch set to determine first further optimized values for each weight associated with a link comprising the selected slice, wherein the first further optimized values are also determined based on the first initial optimized values.

A parallel-computing apparatus configured to optimize a predictive model using training data and knowledge of the placement of parameters within the predictive model, the parallel computing apparatus comprising:

• a controller and a plurality of processing elements communicably coupled to the controller;
• and
• at least one memory storing computer-executable instructions that, when executed by the controller, configure the processing elements to:
• generate a set of slices comprising a predictive model topology wherein:
• the topology comprises a plurality of input nodes, a plurality of output nodes, and a plurality of paths connecting the input nodes to the output nodes;
• each path comprises one or more links associated with respective weights;
• the set of slices comprises a primary slice comprising the topology, a first layer of slices comprising respective subsets of the primary slice, and a second layer of slices comprising respective subsets of the slices comprising the first layer;
• and
• perform first-epoch optimization of the predictive model, comprising:
• dividing the training data into a plurality of batches comprising a first batch set;
• for each slice selected from the second layer, training the selected slice using a first group of batches comprising the first batch set to determine first initial optimized values for each weight associated with a link comprising the selected slice;
• and
• for each slice selected from the first layer, training the selected slice using a second group of batches comprising the first batch set to determine first further optimized values for each weight associated with a link comprising the selected slice, wherein the first further optimized values are also determined based on the first initial optimized values.

A non-transitory, computer-readable medium storing computer-executable instructions that, when executed by a controller of a parallel-computing apparatus, configure a plurality of processing elements comprising the parallel-computing apparatus to:

• generate a set of slices comprising a predictive model topology wherein:
• the topology comprises a plurality of input nodes, a plurality of output nodes, and a plurality of paths connecting the input nodes to the output nodes;
• each path comprises one or more links associated with respective weights;
• the set of slices comprises a primary slice comprising the topology, a first layer of slices comprising respective subsets of the primary slice, and a second layer of slices comprising respective subsets of the slices comprising the first layer;
• and
• perform first-epoch optimization of the predictive model, comprising:
• dividing the training data into a plurality of batches comprising a first batch set;
• for each slice selected from the second layer, training the selected slice using a first group of batches comprising the first batch set to determine first initial optimized values for each weight associated with a link comprising the selected slice;
• and
• for each slice selected from the first layer, training the selected slice using a second group of batches comprising the first batch set to determine first further optimized values for each weight associated with a link comprising the selected slice, wherein the first further optimized values are also determined based on the first initial optimized values.