Systems and methods for reducing idleness in a machine-learning training system using data echoing

A computing system configured to perform a training process to train a machine-learned model based on a set training data that comprises a plurality of training examples, the computing system comprising:

• a computer-readable storage disk configured to store the plurality of training examples;
• a first set of hardware components configured to execute a first set of training operations to access the plurality of training examples from the computer-readable storage disk, prepare the plurality of training examples for use in training the machine-learned model, and store the prepared training examples at an intermediate data buffer;
• the intermediate data buffer configured to store the prepared training examples;
• and
• a second set of hardware components configured to execute a second set of training operations to access the prepared training examples from the intermediate data buffer and to use the prepared training examples to train the machine-learned model;
• wherein the second set of hardware components are configured to perform one or more repeat iterations in which the second set of hardware components repeatedly access at least one of the prepared training examples from the intermediate data buffer and repeatedly use the at least one of the prepared training examples to train the machine-learned model;
• and
• wherein a rate at which the second set of hardware components are configured to perform the one or more repeat iterations is based at least in part on an echo factor that is based at least in part on a comparison of a first computational time associated with performance of the first set of training operations by the first set of hardware components to a second computational time associated with performance of the second set of training operations by the second set of hardware components.