# US 2022/0044149 A1 · Techniques for generating machine learning trained models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044149 A1
- Document ID: US20220044149A1
- Status: Published application
- Kind: A1
- Application number: 17/391,291
- Filed: 2021-08-02
- Published: 2022-02-10
- Earliest claimed: 2020-08-05
- Assignees: Mobileye Vision Technologies Ltd.
- Applicants: Mobileye Vision Technologies Ltd.
- Inventors: Chaim Rand; Aaron Siegel; Amit Weizner
- Web page: https://patentgrep.com/patent/US20220044149A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques are disclosed for the implementation of machine learning model training utilities to generate models for advanced driving assistance system (ADAS), driving assistance, and/or automated vehicle (AV) systems. The techniques described herein may be implemented in conjunction with the utilization of open source and cloud-based machine learning training utilities to generate machine learning trained models. One example of such an open source solution includes TensorFlow, which is a free and open-source software library for dataflow and differentiable programming across a range of tasks. TensorFlow may be used in conjunction with many different types of machine learning utilities.

## Claims

### Claim 1

A machine learning model training system, comprising:
one or more processors; and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
receive labeled training data from a data storage;
preprocess the labeled training data to generate training loop data; and
perform, via a machine learning training loop, training and evaluation of the training loop data in accordance with a defined model loss function to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
wherein the model loss function receives a plurality of tensors associated with a set of labels of the labeled training data and provides model loss function outputs,
wherein the machine learning training loop (i) flattens and concatenates the set of labels to generate a combined input tensor, and (ii) flattens and concatenates the model loss function outputs to generate a combined output tensor, and
wherein the model loss function uses the combined input tensor and the combined output tensor to generate the machine learning trained model.

### Claim 2 (depends on 1)

The machine learning model training system, of claim 1, wherein the one or more processors are configured to preprocess the labeled training data by combining the set of labels of the labeled training data into a single label.

### Claim 3 (depends on 2)

The machine learning model training system of claim 2, wherein the single label of the combined set of labels of the labeled training data have the same name as the combined output tensor.

### Claim 4 (depends on 1)

The machine learning model training system of claim 1, wherein the one or more processors are configured, when executing the instructions stored in the memory, to perform an additional preprocessing using the model loss function to split the combined input tensors and the combined output tensor tensors back into respective individual tensors.

### Claim 5 (depends on 1)

The machine learning model training system of claim 1, wherein the model loss function is represented as a graph having a plurality of layers that include a tf.keras.layers.flatten layer and a tf.keras.layers concatenate layer.

### Claim 6 (depends on 1)

The machine learning model training system of claim 1, wherein the model loss function compromises a TensorFlow Keras loss function.

### Claim 7

A machine learning model training system, comprising:
one or more processors; and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
receive labeled training data from a data storage storing a training dataset;
preprocess the labeled training data to generate training loop data; and
perform, via a machine learning training loop, training and evaluation of the training loop data in accordance with a model loss function to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
wherein the machine learning trained model includes a plurality of layers, the plurality of layers including a loss calculation layer configured to perform a loss calculation in accordance with the model loss function such that the machine learning trained model outputs a result of the loss calculation.

### Claim 8 (depends on 7)

The machine learning model training system of claim 7, wherein the result of the loss calculation are provided by the loss calculation layer as scalar values.

### Claim 9 (depends on 7)

The machine learning model training system of claim 7, wherein labeled training data includes a dummy loss target for the model loss function.

### Claim 10 (depends on 7)

The machine learning model training system of claim 7, wherein the one or more processors are configured to, when executing the instructions stored in the memory:
store each one of a plurality of labels used by the machine learning trained model in the data storage as graph input features; and
relocate each one of the plurality of labels to a dictionary of features in the dataset stored in the data storage.

### Claim 11 (depends on 7)

The machine learning model training system of claim 7, wherein the model loss function comprises a TensorFlow Keras loss function.

### Claim 12

A machine learning model training system, comprising:
one or more processors; and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
receive labeled training data from a data storage;
preprocess the labeled training data to generate training loop data; and
execute a plurality of training steps as part of a machine learning training loop that utilizes the training loop data to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
wherein the machine learning training loop uses an iteratively-executed training function that stores, at each one of the plurality of training steps, data features and labels used by the machine learning trained model, and
wherein the iteratively-executed training function is configured, in response to detecting an error corresponding to model gradients being invalid at a respective one of the plurality of training steps, to stop execution of the machine learning training loop and to store, in the data storage, the data features, labels, and a state of the machine learning trained model corresponding to a respective one of the plurality of training steps at which the error was detected.

### Claim 13 (depends on 12)

The machine learning model training system of claim 12, wherein the iteratively-executed training function is configured to detect the error by comparing the model gradients at each respective one of the plurality of training steps to a predetermined value.

### Claim 14 (depends on 13)

The machine learning model training system of claim 13, wherein the model gradients comprise gradient values, and
wherein the iteratively-executed training function is configured to compare the gradient values at each respective one of the plurality of training steps to the predetermined value prior to applying the gradient values to model weights used in accordance with the machine learning training loop.

### Claim 15 (depends on 14)

The machine learning model training system of claim 14, wherein the predetermined value is identified with a Not a Number (NaN) value.

### Claim 16 (depends on 12)

The machine learning model training system of claim 12, wherein the state of the machine learning trained model stored in the data storage comprises model weights corresponding to a respective one of the plurality of training steps at which the invalid the model gradients were detected.

### Claim 17 (depends on 12)

The machine learning model training system of claim 12, wherein the iteratively-executed training function comprises a tf.keras.models.Model object used in accordance with Tensorflow.

### Claim 18 (depends on 17)

The machine learning model training system of claim 17, wherein the tf.keras.models.Model object comprises a class that defines a Boolean flag to signal to the iteratively-executed training function whether the error was detected.

### Claim 19

A machine learning model training system, comprising:
one or more processors; and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
receive labeled training data from a data storage;
preprocess the labeled training data to generate training loop data; and
execute a machine learning training loop that utilizes the training loop data to generate a machine learning trained model that enables machine vision to recognize and classify objects included in a road scene,
wherein the machine learning trained model comprises a plurality of layers, the plurality of layers including a capture layer interleaved between a first layer and a second layer of the plurality of layers and configured to capture intermediate tensors that are output by the first layer and input to the second layer, and
wherein the capture layer causes the one or more processors to store the intermediate tensors in the data storage.

### Claim 20 (depends on 19)

The machine learning model training system of claim 19, wherein the intermediate tensors are multi-dimensional arrays of a uniform type.

### Claim 21 (depends on 19)

The machine learning model training system of claim 19, wherein the capture layer is configured to cause the one or more processors to store the intermediate tensors as an internal non-trainable weight variable.

### Claim 22 (depends on 19)

The machine learning model training system of claim 19, wherein the machine learning training loop utilizes the training loop data to generate the machine learning trained model in accordance with a model loss function, and
wherein the capture layer causes the one or more processors to store, as the intermediate tensors in the data storage, tensors identified with the model loss function.

### Claim 23 (depends on 19)

The machine learning model training system of claim 19, wherein the capture layer does not perform calculations in accordance with the machine learning trained model.

### Claim 24 (depends on 19)

The machine learning model training system of claim 19, wherein the capture layer comprises a TensorFlow Keras layer.
