# US 12,112,517 B2 · Diagnostic tool for deep learning similarity models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,517 B2
- Document ID: US12112517B2
- Status: Granted
- Kind: B2
- Application number: 18/353,045
- Filed: 2023-07-14
- Published: 2024-01-25
- Granted: 2024-10-08
- Earliest claimed: 2020-09-25
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Oren Barkan; Omri Armstrong; Ori Katz; Noam Koenigstein
- Web page: https://patentgrep.com/patent/US12112517B2

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations}
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7784: {based on feedback from supervisors}
- G06N 3/045: Combinations of networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A diagnostic tool for deep learning similarity models and image classifiers provides valuable insight into neural network decision-making. A disclosed solution generates a saliency map by: receiving a test image; determining, with an image classifier, an image classification of the test image; determining, for the test image, a first activation map for at least one model layer using the determined image classification; determining, for the test image, a first gradient map for the at least one model layer using the determined image classification; and generating a first saliency map as an element-wise function of the first activation map and the first gradient map.

## Claims

### Claim 1

A system comprising:
a processor;
a computer-readable medium storing instructions;
the instructions stored on the computer-readable medium executed by the processor to:
receive a test image;
determine, with an image classifier, an image classification of the test image;
determine, for the test image, a first activation map for at least one model layer using the determined image classification;
determine, for the test image, a first gradient map for the at least one model layer using the determined image classification; and
generate a first saliency map as a first element-wise function of the first activation map and the first gradient map.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions are further operative to:
determine a region of interest (ROI) in the first saliency map;
crop the test image to an area corresponding to the determined ROI; and
determine a refined image classification.

### Claim 3 (depends on 1)

The system of claim 1, wherein the at least one model layer is prior to a penultimate hidden layer.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first element-wise function of the first activation map and the first gradient map comprises a Hadamard product.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions are further operative to:
output the first saliency map as a heatmap image.

### Claim 6 (depends on 1)

The system of claim 1, wherein the instructions are further operative to:
determine, for the test image, a second activation map using the determined image classification;
determine, for the test image, a second gradient map using the determined image classification;
generate a second saliency map as a second element-wise function of the second activation map and the second gradient map; and
generate a composite saliency map as a third element-wise function of the first saliency map and the second saliency map.

### Claim 7 (depends on 6)

The system of claim 6, wherein the instructions are further operative to:
output the composite saliency map as a heatmap image.

### Claim 8

A method comprising:
receiving a test image;
determining, with an image classifier, an image classification of the test image;
determining, for the test image, a first activation map for at least one model layer using the determined image classification;
determining, for the test image, a first gradient map for the at least one model layer using the determined image classification; and
generating a first saliency map as a first element-wise function of the first activation map and the first gradient map.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
determining a region of interest (ROI) in the first saliency map;
cropping the test image to an area corresponding to the determined ROI; and
determining a refined image classification.

### Claim 10 (depends on 8)

The method of claim 8, wherein the at least one model layer is prior to a penultimate hidden layer.

### Claim 11 (depends on 8)

The method of claim 8, wherein the first element-wise function of the first activation map and the first gradient map comprises a Hadamard product.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
outputting the first saliency map as a heatmap image.

### Claim 13 (depends on 8)

The method of claim 8, further comprising:
determining, for the test image, a second activation map using the determined image classification;
determining, for the test image, a second gradient map using the determined image classification;
generating a second saliency map as a second element-wise function of the second activation map and the second gradient map; and
generating a composite saliency map as a third element-wise function of the first saliency map and the second saliency map.

### Claim 14 (depends on 3)

The method of claim 3, further comprising:
outputting the composite saliency map as a heatmap image.

### Claim 15

One or more computer storage devices having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:
receiving a test image;
determining, with an image classifier, an image classification of the test image;
determining, for the test image, a first activation map for at least one model layer using the determined image classification;
determining, for the test image, a first gradient map for the at least one model layer using the determined image classification; and
generating a first saliency map as a first element-wise function of the first activation map and the first gradient map.

### Claim 16 (depends on 15)

The one or more computer storage devices of claim 15, wherein the operations further comprise:
determining a region of interest (ROI) in the first saliency map;
cropping the test image to an area corresponding to the determined ROI; and
determining a refined image classification.

### Claim 17 (depends on 15)

The one or more computer storage devices of claim 15, wherein the at least one model layer is prior to a penultimate hidden layer.

### Claim 18 (depends on 15)

The one or more computer storage devices of claim 15, wherein the first element-wise function of the first activation map and the first gradient map comprises a Hadamard product.

### Claim 19 (depends on 15)

The one or more computer storage devices of claim 15, wherein the operations further comprise:
outputting the first saliency map as a heatmap image.

### Claim 20 (depends on 15)

The one or more computer storage devices of claim 15, wherein the operations further comprise:
determining, for the test image, a second activation map using the determined image classification;
determining, for the test image, a second gradient map using the determined image classification;
generating a second saliency map as a second element-wise function of the second activation map and the second gradient map; and
generating a composite saliency map as a third element-wise function of the first saliency map and the second saliency map.