# US 2022/0092366 A1 · Region metrics for class balancing in machine learning systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092366 A1
- Document ID: US20220092366A1
- Status: Published application
- Kind: A1
- Application number: 17/478,177
- Filed: 2021-09-17
- Published: 2022-03-24
- Earliest claimed: 2020-09-24
- Assignees: SRI International
- Applicants: SRI International
- Inventors: Han-Pang Chiu; Junjiao Tian; Zachary Seymour; Niluthpol C. Mithun
- Web page: https://patentgrep.com/patent/US20220092366A1

## CPC

- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2193: {based on specific statistical tests}
- G06F 18/2431: Multiple classes
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are disclosed for an image understanding system comprising a machine learning system that applies a machine learning model to perform image understanding of each pixel of an image, the pixel labeled with a class, to determine an estimated class to which the pixel belongs. The machine learning system determines, based on the classes with which the pixels are labeled and the estimated classes, a cross entropy loss of each class. The machine learning system determines, based on one or more region metrics, a weight for each class and applies the weight to the cross entropy loss of each class to obtain a weighted cross entropy loss. The machine learning system updates the machine learning model with the weighted cross entropy loss to improve a performance metric of the machine learning model for each class.

## Claims

### Claim 1

An image understanding system comprising:
an input device configured to receive training data comprising an image comprising a plurality of pixels, each pixel of the plurality of pixels labeled with a class of a plurality of classes; and
a computation engine comprising processing circuitry for executing a machine learning system, wherein the machine learning system is configured to:
apply a machine learning model to perform image understanding of each pixel of the plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs;
determine, based on the classes with which the plurality of pixels are labeled and the estimated classes of the plurality of pixels, a cross entropy loss of each class of the plurality of classes;
determine, based on one or more region metrics, a weight for each class of the plurality of classes;
apply the weight for each class of the plurality of classes to the cross entropy loss of each class of the plurality of classes to obtain a weighted cross entropy loss of each class of the plurality of classes; and
update the machine learning model with the weighted cross entropy loss of each class of the plurality of classes to improve a performance metric of the machine learning model for each class of the plurality of classes.

### Claim 2 (depends on 1)

The system of claim 1, wherein the machine learning system is configured to apply the updated machine learning model to perform image understanding, determine the cross entropy loss of each class of the plurality of classes, determine the weight for each class of the plurality of classes, apply the weight to obtain the weighted cross entropy loss of each class of the plurality of classes, and update the updated machine learning model with the weighted cross entropy loss of each class of the plurality of classes.

### Claim 3 (depends on 2)

The system of claim 2, wherein to iteratively determine the weight for each class of the plurality of classes, the machine learning system is configured to:
reduce the weight for each class of the plurality of classes as the performance metric for the class increases; and
increase the weight for each class of the plurality of classes as the performance metric for the class decreases.

### Claim 4 (depends on 1)

The system of claim 1,
wherein the input device is configured to receive a second image comprising a second plurality of pixels,
wherein the machine learning system is configured to apply the machine learning model to perform image understanding of each pixel of the second plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs, and
wherein the system further comprises an output device configured to output, for display to a user, an indication of the estimated class of the plurality of classes to which each pixel of the second plurality of pixels belongs.

### Claim 5 (depends on 1)

The system of claim 1,
wherein the input device is configured to receive a second image comprising a second plurality of pixels,
wherein the machine learning system is configured to apply the machine learning model to perform image understanding of each pixel of the second plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs, and
wherein the system further comprises an output device configured to output, based on the estimated class of the plurality of classes to which each pixel of the second plurality of pixels belongs, navigation information for use by one or more of a moving vehicle or a mobile platform.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more region metrics are computed according to a Recall loss function.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more region metrics are computed according to at least one of:
a Precision loss function;
a Dice loss function;
a Jaccard loss function;
an F1 loss function; or
a Tversky loss function.

### Claim 8 (depends on 1)

The system of claim 1, wherein to apply the machine learning model to perform image understanding of each pixel of the plurality of pixels to determine the estimated class of the plurality of classes to which the pixel belongs, the machine learning system is configured to apply the machine learning model to perform at least one of:
semantic segmentation of each pixel of the plurality of pixels;
object detection of an object represented in the plurality of pixels;
object recognition of the object represented in the plurality of pixels; or
image recognition of the image comprising the plurality of pixels.

### Claim 9 (depends on 1)

The system of claim 1, wherein the performance metric of the machine learning model comprises at least one of a mean accuracy or a mean Intersection Over Union (IOU).

### Claim 10 (depends on 1)

The system of claim 1, wherein the plurality of classes comprises an imbalanced plurality of classes such that a first class of the plurality of classes is represented in the training data more frequently than a second class of the plurality of classes is represented in the training data.

### Claim 11 (depends on 1)

The system of claim 1, wherein the cross entropy loss of each class of the plurality of classes comprises a probability, for each class of the plurality of classes, that an estimated class corresponds to a class of a label applied to a pixel, an object or an image of the training data.

### Claim 12

A method for image understanding comprising:
receiving, by an input device, training data comprising an image comprising a plurality of pixels, each pixel of the plurality of pixels labeled with a class of a plurality of classes;
applying, by a machine learning system of a computation engine executed by processing circuitry, a machine learning model to perform image understanding of each pixel of the plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs;
determining, by the machine learning system and based on the classes with which the plurality of pixels are labeled and the estimated classes of the plurality of pixels, a cross entropy loss of each class of the plurality of classes;
determining, by the machine learning system and based on one or more region metrics, a weight for each class of the plurality of classes;
applying, by the machine learning system, the weight for each class of the plurality of classes to the cross entropy loss of each class of the plurality of classes to obtain a weighted cross entropy loss of each class of the plurality of classes; and
updating, by the machine learning system, the machine learning model with the weighted cross entropy loss of each class of the plurality of classes to improve a performance metric of the machine learning model for each class of the plurality of classes.

### Claim 13 (depends on 12)

The method of claim 12, further comprising applying the updated machine learning model to perform image understanding, determining the cross entropy loss of each class of the plurality of classes, determining the weight for each class of the plurality of classes, applying the weight to obtain the weighted cross entropy loss of each class of the plurality of classes, and updating the updated machine learning model with the weighted cross entropy loss of each class of the plurality of classes.

### Claim 14 (depends on 13)

The method of claim 13, wherein iteratively determining the weight for each class of the plurality of classes comprises:
reducing the weight for each class of the plurality of classes as the performance metric for the class increases; and
increasing the weight for each class of the plurality of classes as the performance metric for the class decreases.

### Claim 15 (depends on 12)

The method of claim 12, further comprising:
receiving, by the input device, a second image comprising a second plurality of pixels;
applying, by the machine learning system, the machine learning model to perform image understanding of each pixel of the second plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs; and
outputting, by an output device and for display to a user, an indication of the estimated class of the plurality of classes to which each pixel of the second plurality of pixels belongs.

### Claim 16 (depends on 12)

The method of claim 12, wherein the one or more region metrics are computed according to a Recall loss function.

### Claim 17 (depends on 12)

The method of claim 12, wherein the one or more region metrics are computed according to at least one of:
a Precision loss function;
a Dice loss function;
a Jaccard loss function;
an F1 loss function; or
a Tversky loss function.

### Claim 18 (depends on 12)

The method of claim 12, wherein the performance metric of the machine learning model comprises at least one of a mean accuracy or a mean Intersection Over Union (IOU).

### Claim 19 (depends on 12)

The method of claim 12, wherein the plurality of classes comprises an imbalanced plurality of classes such that a first class of the plurality of classes is represented in the training data more frequently than a second class of the plurality of classes is represented in the training data.

### Claim 20

A non-transitory, computer-readable medium comprising instructions for causing processing circuitry of an image understanding system to:
receive training data comprising an image comprising a plurality of pixels, each pixel of the plurality of pixels labeled with a class of a plurality of classes; and
execute a machine learning system configured to:
apply a machine learning model to perform image understanding of each pixel of the plurality of pixels to determine an estimated class of the plurality of classes to which the pixel belongs;
determine, based on the classes with which the plurality of pixels are labeled and the estimated classes of the plurality of pixels, a cross entropy loss of each class of the plurality of classes;
determine, based on one or more region metrics, a weight for each class of the plurality of classes;
apply the weight for each class of the plurality of classes to the cross entropy loss of each class of the plurality of classes to obtain a weighted cross entropy loss of each class of the plurality of classes; and
update the machine learning model with the weighted cross entropy loss of each class of the plurality of classes to improve a performance metric of the machine learning model for each class of the plurality of classes.
