# US 2021/0241034 A1 · Method of and system for generating training images for instance segmentation machine learning algorithm

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241034 A1
- Document ID: US20210241034A1
- Status: Published application
- Kind: A1
- Application number: 16/778,597
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Element Al Inc.
- Applicants: Element Al Inc.
- Inventors: Issam Hadj Laradji; David Vazquez Bermudez
- Web page: https://patentgrep.com/patent/US20210241034A1

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/24: Classification techniques
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20156: Automatic seed setting
- G06T 2207/20164: Salient point detection; Corner detection
- G06V 10/759: {Region-based matching}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and a system for generating training images for training an instance segmentation machine learning algorithm (MLA). A set of image-level labelled images are received, where a given image is labelled with a label indicative of a presence of an object having an object class in the image. A classification MLA detects the object having the object class in each image. A class activation map (CAM) indicative of discriminative regions used by the classification MLA for detecting the object in each image is generated. A region proposal MLA is used to generate region proposals for each image. A pseudo mask of the respective object is generated based on the region proposals and the CAM, where a pseudo mask is indicative of pixels corresponding to the respective object class. The pseudo masks are used as a label with the image-level labelled images for training the instance segmentation MLA.

## Claims

### Claim 1

A method for generating a set of training images for an instance segmentation machine learning algorithm (MLA), the method being executed by a processor, the processor having access to:
a classification MLA having been trained to detect objects in an image and generate a class activation map (CAM) indicative of discriminative regions used for detecting the objects,
a region proposal MLA having been trained to generate region proposals from an image,
the method comprising:
receiving a set of image-level labelled images, each image-level labelled image comprising an object class label indicative of a presence of a respective object having a respective object class in the image;
detecting, using the classification MLA, in each image-level labelled image, the respective object having the respective object class;
determining, using the classification MLA, for each image-level labelled image of the set of image-level labelled images, a respective CAM indicative of discriminative regions used by the classification MLA to detect the respective object class;
generating, using the region proposal MLA, for each image-level labelled image of the set of image-level labelled images, a respective set of region proposals, each region proposal comprising a respective potential object;
generating, for each image-level labelled image of the set of image-level labelled images, based on the respective CAM and the respective set of region proposals, a respective pseudo mask of the respective object indicative of pixels in the image-level labelled image corresponding to the respective object class; and
generating a set of training images to be provided for training an instance segmentation MLA, each training image comprising:
a respective object class of the respective object, and
the respective pseudo mask of the respective object having the respective object class.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
training the instance segmentation MLA on the set of training images by using the respective pseudo mask as a target for generating a predicted mask for an object class of an object in a new image.

### Claim 3 (depends on 1)

The method of claim 1, wherein
the method further comprises, prior to the generating the pseudo mask for each image-level labelled image:
generating, by the classification MLA, a respective peak response map (PRM) for each respective CAM by determining a respective set of peaks indicative of local maximas in the respective CAM; and wherein
the generating the pseudo mask is further based on the respective PRM.

### Claim 4 (depends on 3)

The method of claim 3, wherein the respective set of peaks are indicative of approximate locations of objects having the respective object class.

### Claim 5 (depends on 4)

The method of claim 4, wherein the generating the respective pseudo mask for the respective object class comprises:
selecting at least one region proposal of the set of region proposals intersecting with a peak of the respective set of peaks in the PRM to obtain the respective pseudo mask.

### Claim 6 (depends on 5)

The method of claim 5, wherein
each region proposal of the respective set of region proposals is associated with a respective objectness score indicative of a probability of the region proposal comprising a respective object; and wherein
the selecting the at least one region proposal is based on the respective objectness score.

### Claim 7 (depends on 3)

The method of claim 3, wherein the classification MLA comprises a convolutional neural network, a fully connected layer (FCL), and a peak stimulation layer.

### Claim 8 (depends on 1)

The method of claim 1, wherein the instance segmentation MLA comprises a convolutional neural network.

### Claim 9

A method for training an instance segmentation machine learning algorithm (MLA), the method being executed by a processor, the processor having access to:
a classification MLA having been trained to detect objects in an image and generate a class activation map (CAM) indicative of discriminative regions used for detecting the objects,
a region proposal MLA having been trained to generate region proposals from an image, and
the instance segmentation MLA,
the method comprising:
receiving a set of image-level labelled images, each image-level labelled image comprising a respective object class label indicative of a presence of a respective object having a respective object class in the image;
detecting, using the classification MLA, in each image-level labelled image, the respective object having the respective object class, the detecting comprising generating a respective CAM indicative of discriminative regions used by the classification MLA to detect the respective object class;
generating, using the region proposal MLA, for each image-level labelled image of the set of image-level labelled images, a respective set of region proposals, each region proposal comprising a respective potential object;
generating, for each image-level labelled image of the set of image-level labelled images, based on the respective CAM and the respective set of region proposals, a respective pseudo mask of the respective object indicative of pixels in the image-level labelled image corresponding to the respective object class; and
training an instance segmentation MLA on the set of image-level labelled images and the respective pseudo masks by using the respective pseudo mask having the respective object class label as a target.

### Claim 10 (depends on 9)

The method of claim 9, wherein
the detecting further comprises:
generating, by the classification MLA, a respective peak response map (PRM) for each respective CAM by determining a respective set of peaks indicative of local maximas in the respective CAM; and wherein
the generating the pseudo mask is further based on the respective PRM.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
receiving a new image, the new image not being included in the set of image-level labelled images;
generating, by the instance segmentation MLA, a set of image features; and
detecting, by the instance segmentation MLA, based on the set of image features, an object, the object having an object class, the detecting comprising classifying a set of pixels in the image as belonging to the object class of the object to obtain a predicted mask of the object.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
generating, using the region proposal MLA, a set of region proposals for the new image; and
generating, based on the set of region proposals and the predicted mask of the object, a refined predicted mask.

### Claim 13

A system for generating a set of training images for training an instance segmentation machine learning algorithm (MLA), the system comprising a processor, the processor having access to:
a classification MLA having been trained to detect objects in an image and generate a class activation map (CAM) indicative of discriminative regions used for detecting the objects,
a region proposal MLA having been trained to generate region proposals from an image,
the processor being operatively connected to a non-transitory storage medium comprising instructions;
the processor, upon executing the instructions, being configured for:
receiving a set of image-level labelled images, each image-level labelled image comprising an object class label indicative of a presence of a respective object having a respective object class in the image;
detecting, using the classification MLA, in each image-level labelled image, the respective object having the respective object class;
determining, using the classification MLA, for each image-level labelled image of the set of image-level labelled images, a respective CAM indicative of discriminative regions used by the classification MLA to detect the respective object class;
generating, using the region proposal MLA, for each image-level labelled image of the set of image-level labelled images, a respective set of region proposals, each region proposal comprising a respective potential object;
generating, for each image-level labelled image of the set of image-level labelled images, based on the respective CAM and the respective set of region proposals, a respective pseudo mask of the respective object indicative of pixels in the image-level labelled image corresponding to the respective object class; and
generating a set of training images to be provided for training an instance segmentation MLA, each training image comprising:
a respective object class of the respective object, and
the respective pseudo mask of the respective object having the respective object class.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor is further configured for:
training the instance segmentation MLA on the set of training images by using the respective pseudo mask as a target for generating a predicted mask for an object class of an object in a new image.

### Claim 15 (depends on 13)

The system of claim 13, wherein
the processor is further configured for, prior to the generating the pseudo mask for each image-level labelled image:
generating, by the classification MLA, a respective peak response map (PRM) for each respective CAM by determining a respective set of peaks indicative of local maximas in the respective CAM; and wherein
the generating the pseudo mask is further based on the respective PRM.

### Claim 16 (depends on 15)

The system of claim 15, wherein the respective set of peaks are indicative of approximate locations of objects having the respective object class.

### Claim 17 (depends on 16)

The system of claim 16, wherein the generating the respective pseudo mask for the respective object class comprises:
selecting at least one region proposal of the set of region proposals intersecting with a peak of the respective set of peaks in the PRM to obtain the respective pseudo mask.

### Claim 18 (depends on 17)

The system of claim 17, wherein
each region proposal of the respective set of region proposals is associated with a respective objectness score indicative of a probability of the region proposal comprising a respective object; and wherein
the selecting the at least one region proposal is based on the respective objectness score.

### Claim 19 (depends on 15)

The system of claim 15, wherein the classification MLA comprises a convolutional neural network, a fully connected layer (FCL), and a peak stimulation layer.

### Claim 20 (depends on 15)

The system of claim 15, wherein the instance segmentation MLA comprises a convolutional neural network.
