Error checking for machine learning hardware accelerators

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving data representing a machine learning training workload for training a neural network;
• and
• executing the machine learning training workload across a plurality of hardware devices, the executing comprising, at each of at least a subset of the plurality of hardware devices, repeatedly performing training operations comprising:
• performing one or more computation operations to generate local output data at a corresponding time point;
• generating, based at least on the local output data, a first checksum for the corresponding time point for detecting errors induced by hardware faults;
• storing the first checksum in a storage device;
• and
• transmitting the local output data from the hardware device to another hardware device over a network.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving data representing a machine learning training workload for training a neural network;
• and
• executing the machine learning training workload across a plurality of hardware devices, the executing comprising, at each of at least a subset of the plurality of hardware devices, repeatedly performing training operations comprising:
• performing one or more computation operations to generate local output data at a corresponding time point;
• generating, based at least on the local output data, a first checksum for the corresponding time point for detecting errors induced by hardware faults;
• storing the first checksum in a storage device;
• and
• transmitting the local output data from the hardware device to another hardware device over a network.