# US 2023/0410469 A1 · Systems and methods for image classification using a neural network combined with a correlation structure

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410469 A1
- Document ID: US20230410469A1
- Status: Published application
- Kind: A1
- Application number: 17/807,550
- Filed: 2022-06-17
- Published: 2023-12-21
- Earliest claimed: 2022-06-17
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Thomas Muehlenstaedt; Jelena Frtunikj
- Web page: https://patentgrep.com/patent/US20230410469A1

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for performing image classification are disclosed. The methods include, by a processor: receiving an input image, generating a label prediction corresponding to the input image using a trained neural network, generating a correlation structure based on a comparison of the input image with each of a plurality of reference images, and generating an updated label prediction corresponding to the input image using the label prediction and the correlation structure.

## Claims

### Claim 1

A method of performing image classification, the method comprising, by a processor:
receiving an input image;
generating, using a trained neural network, a label prediction corresponding to the input image;
generating, based on a comparison of the input image with each of a plurality of reference images, a correlation structure; and
generating, using the label prediction and the correlation structure, an updated label prediction corresponding to the input image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising using the updated label prediction for controlling navigation of an autonomous vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the correlation structure comprises computing a distance between the input image and each of the plurality of reference images, the distance being associated with one or more color channels.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the updated label prediction corresponding to the input image comprises identifying the label prediction as the updated label prediction in response to determining that there exists a correlation between the input image and each of the plurality of reference images that is less than a threshold.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the updated label prediction corresponding to the input image comprises identifying a correct label associated with a reference image as the updated label prediction in response to determining that there exists a high correlation between the input image and the reference image.

### Claim 6 (depends on 1)

The method of claim 1, further comprising generating a reference dataset comprising the plurality of reference images, wherein at least one of the plurality of reference images comprises an adversarial image or an image associated with an edge case.

### Claim 7 (depends on 6)

The method of claim 6, further comprising generating the reference dataset to include a correct label prediction associated with each of the plurality of reference images.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the correlation structure comprises computing a distance between the input image and each of the plurality of reference images by comparing a feature map of the input image and a reference feature map of that reference image, the feature map and the reference feature map obtained from a layer of the trained neural network.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the correlation structure comprises:
identifying a reference image of the plurality of reference images that has a highest correlation with the input image; and
identifying a correct label associated with the identified reference image as the updated label prediction.

### Claim 10 (depends on 1)

The method of claim 1, wherein the correlation structure is a Gaussian process correlation structure.

### Claim 11

A system for performing image classification, the system comprising:
a processor; and
a non-transitory computer-readable medium comprising programming instructions that when executed by the processor will cause the processor to:
receive an input image,
generate, using a trained neural network, a label prediction corresponding to the input image,
generate, based on a comparison of the input image with each of a plurality of reference images, a correlation structure, and
generate, using the label prediction and the correlation structure, an updated label prediction corresponding to the input image.

### Claim 12 (depends on 11)

The system of claim 11, further comprising programming instructions that when executed by the processor will cause the processor to use the updated label prediction for controlling navigation of an autonomous vehicle.

### Claim 13 (depends on 11)

The system of claim 11, wherein the programming instructions that when executed by the processor will cause the processor to generate the correlation structure comprise programming instructions to cause the processor to compute a distance between the input image and each of the plurality of reference images, the distance being associated with one or more color channels.

### Claim 14 (depends on 11)

The system of claim 11, wherein the programming instructions that when executed by the processor will cause the processor to generate the updated label prediction corresponding to the input image comprise programming instructions to cause the processor to identify the label prediction as the updated label prediction in response to determining that there exists a correlation between the input image and each of the plurality of reference images that is less than a threshold.

### Claim 15 (depends on 11)

The system of claim 11, wherein the programming instructions that when executed by the processor will cause the processor to generate the updated label prediction corresponding to the input image comprise programming instructions to cause the processor to identify a correct label associated with a reference image as the updated label prediction in response to determining that there exists a high correlation between the input image and the reference image.

### Claim 16 (depends on 11)

The system of claim 11, further comprising programming instructions that when executed by the processor will cause the processor to generate a reference dataset comprising the plurality of reference images, wherein at least one of the plurality of reference images comprises an adversarial image or an image associated with an edge case.

### Claim 17 (depends on 16)

The system of claim 16, further comprising programming instructions that when executed by the processor will cause the processor to generate the reference dataset to include a correct label prediction associated with each of the plurality of reference images.

### Claim 18 (depends on 11)

The system of claim 11, wherein the programming instructions that when executed by the processor will cause the processor to generate the correlation structure comprise programming instructions to cause the processor to compute a distance between the input image and each of the plurality of reference images by comparing a feature map of the input image and a reference feature map of that reference image, the feature map and the reference feature map obtained from a layer of the trained neural network.

### Claim 19 (depends on 11)

The system of claim 11, wherein the programming instructions that when executed by the processor will cause the processor to generate the correlation structure comprise programming instructions to cause the processor to:
identify a reference image of the plurality of reference images that has a highest correlation with the input image; and
identify a correct label associated with the identified reference image as the updated label prediction.

### Claim 20

A computer program product comprising a memory and programming instructions that are configured to cause a processor to:
receive an input image;
generate, using a trained neural network, a label prediction corresponding to the input image;
generate, based on a comparison of the input image with each of a plurality of reference images, a correlation structure; and
generate, using the label prediction and the correlation structure, an updated label prediction corresponding to the input image.
