Techniques for generating machine learning trained models

A machine learning model training system, comprising:

• one or more processors;
• and
• a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive labeled training data from a data storage storing a training dataset;
• preprocess the labeled training data to generate training loop data;
• and
• perform, via a machine learning training loop, training and evaluation of the training loop data in accordance with a model loss function to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
• wherein the machine learning trained model includes a plurality of layers, the plurality of layers including a loss calculation layer configured to perform a loss calculation in accordance with the model loss function such that the machine learning trained model outputs a result of the loss calculation.

A machine learning model training system, comprising:

• one or more processors;
• and
• a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive labeled training data from a data storage;
• preprocess the labeled training data to generate training loop data;
• and
• execute a plurality of training steps as part of a machine learning training loop that utilizes the training loop data to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
• wherein the machine learning training loop uses an iteratively-executed training function that stores, at each one of the plurality of training steps, data features and labels used by the machine learning trained model, and
• wherein the iteratively-executed training function is configured, in response to detecting an error corresponding to model gradients being invalid at a respective one of the plurality of training steps, to stop execution of the machine learning training loop and to store, in the data storage, the data features, labels, and a state of the machine learning trained model corresponding to a respective one of the plurality of training steps at which the error was detected.

A machine learning model training system, comprising:

• one or more processors;
• and
• a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive labeled training data from a data storage;
• preprocess the labeled training data to generate training loop data;
• and
• execute a machine learning training loop that utilizes the training loop data to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
• wherein the machine learning trained model comprises a plurality of layers, the plurality of layers including a capture layer interleaved between a first layer and a second layer of the plurality of layers and configured to capture intermediate tensors that are output by the first layer and input to the second layer, and
• wherein the capture layer causes the one or more processors to store the intermediate tensors in the data storage.