# US 2023/0410484 A1 · Anomaly detection using a convolutional neural network and feature based memories

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410484 A1
- Document ID: US20230410484A1
- Status: Published application
- Kind: A1
- Application number: 18/035,942
- Filed: 2022-04-25
- Published: 2023-12-21
- Earliest claimed: 2021-04-26
- Assignees: Jidoka Technologies Private Limited
- Applicants: Jidoka Technologies Private Limited
- Inventors: Sekar UDAYAMURTHY; Mohan NIKAM; Vinodh VENKATESH; Krishna IYENGAR
- Web page: https://patentgrep.com/patent/US20230410484A1

## CPC

- G06V 10/776: Validation; Performance evaluation
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06N 3/094: Adversarial learning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments of the present disclosure discuss a system and a method for anomaly detection in an object. Conventional systems of defect detection require labeled training data. Further, current anomaly detection systems cannot be easily incorporated into factory production systems. Embodiments of the present disclosure address this problem by using a convolutional neural network (CNN) and feature based memories such as neural network based memories for anomaly detection. The method described in the embodiments uses neural network based memories to generate memory feature maps from feature maps extracted at intermediate layers of the CNN. Subsequently, heatmaps are created by calculating feature differences between the feature maps and the memory feature maps. Weighted averaging and post processing is performed on the heatmaps to identify the anomaly in the object.

## Claims

### Claim 1

A method for anomaly detection comprising:
receiving, by a processor, an image of an object (310);
inputting, by the processor, the image to a pre-trained convolutional neural network including a plurality of intermediate layers (320);
extracting, by the processor, feature maps output at the plurality of intermediate layers (330);
inputting, by the processor, the feature maps extracted at the plurality of intermediate layers to pretrained neural network based memories corresponding to the plurality of intermediate layers to generate memory feature maps (340);
creating, by the processor, a two-dimensional heatmap corresponding to each intermediate layer of the plurality of intermediate layers by calculating a feature difference between the feature map extracted at the intermediate layer and the memory feature map generated by the pretrained neural network based memory corresponding to the intermediate layer (350);
resizing, by the processor, the two-dimensional heatmaps to the dimensions of the image (360);
obtaining, by the processor, an anomaly heat map by performing a weighted average of the resized two-dimensional heatmaps (370); and
identifying, by the processor, an anomaly in the object by discarding heatmap intensities below a threshold in the anomaly heatmap (380).

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the image is received from an image acquisition unit (120).

### Claim 3 (depends on 2)

The method as claimed in claim 2, wherein the image acquisition unit (120) comprises a plurality of cameras.

### Claim 4 (depends on 1)

The method as claimed in claim 1, wherein the pretrained neural network based memories include an auto-encoder.

### Claim 5 (depends on 1)

The method as claimed in claim 1, wherein the pretrained neural network based memories includes a generative adversarial network.

### Claim 6 (depends on 1)

The method as claimed in claim 1, wherein the threshold is calculated using a business logic.

### Claim 7 (depends on 1)

The method as claimed in claim 1, wherein the pretrained neural network based memory corresponding to each intermediate layer is trained using feature maps extracted at the layer when images of a training set are input to the pretrained convolutional neural network.

### Claim 8 (depends on 1)

The method as claimed in claim 1, comprising classifying the image based on the identified anomaly.

### Claim 9 (depends on 1)

The method as claimed in claim 1, wherein the object is classified as anomalous based on the identified anomaly.

### Claim 10

A system (100) for anomaly detection comprising:
a memory (144); and
at least one processor (142) communicatively coupled to the memory (144), wherein the at least one processor (142) is configured to execute instructions stored in the memory to:
receive an image of an object (310);
input the image to a pre-trained convolutional neural network including a plurality of intermediate layers (320);
extract feature maps output at the plurality of intermediate layers (330);
input the feature maps extracted at the plurality of intermediate layers to pretrained neural network based memories corresponding to the plurality of intermediate layers to generate memory feature maps (340);
create a two-dimensional heatmap corresponding to each intermediate layer of the plurality of intermediate layers by calculating a feature difference between the feature map extracted at the intermediate layer and the memory feature map generated by the pretrained neural network based memory corresponding to the intermediate layer (350);
resize the two-dimensional heatmaps to the dimensions of the image (360);
obtain an anomaly heat map by performing a weighted average of the resized two-dimensional heatmaps (370); and
identify an anomaly in the object by discarding heatmap intensities below a threshold in the anomaly heatmap (380).

### Claim 11 (depends on 10)

The system as claimed in claim 10, wherein the image is received from an image acquisition unit (120).

### Claim 12 (depends on 11)

The system as claimed in claim 11, wherein the image acquisition unit (120) comprises a plurality of cameras.

### Claim 13 (depends on 10)

The system as claimed in claim 10, wherein the pretrained neural network based memories include an auto-encoder.

### Claim 14 (depends on 10)

The system as claimed in claim 10, wherein the pretrained neural network based memories include a generative adversarial network.

### Claim 15 (depends on 10)

The system as claimed in claim 10, wherein the threshold is calculated using a business logic.

### Claim 16 (depends on 10)

The system as claimed in claim 10, wherein the pretrained neural network based memory corresponding to each intermediate layer is trained using feature maps extracted at the layer when images of a training set are input to the pretrained convolutional neural network.

### Claim 17 (depends on 10)

The system as claimed in claim 10, comprising classifying the image based on the identified anomaly.

### Claim 18 (depends on 10)

The system as claimed in claim 10, wherein the object is classified as anomalous based on the identified anomaly.