US 11,308,353 B2 · Classifying digital images in few-shot tasks based on neural networks trained using manifold mixup regularization and self-supervision

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for training a classification neural network to classify digital images in few-shot tasks based on self-supervision and manifold mixup. For example, the disclosed systems can train a feature extractor as part of a base neural network utilizing self-supervision and manifold mixup. Indeed, the disclosed systems can apply manifold mixup regularization over a feature manifold learned via self-supervised training such as rotation training or exemplar training. Based on training the feature extractor, the disclosed systems can also train a classifier to classify digital images into novel classes not present within the base classes used to train the feature extractor.

Claims20

Dependent claims

A computer-implemented method comprising:

identifying a set of labeled digital images corresponding to a plurality of base classes;
performing a step for training a base neural network to classify a digital image into one of the plurality of base classes;
and
performing a step for training a second classifier to classify the digital image into an additional class not present in the plurality of base classes of the set of labeled digital images.

1 more element

A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:

extract features from a digital image using a feature extractor of a base neural network trained to classify the digital image utilizing manifold mixup regularization to modify decision boundaries of a feature manifold generated via self-supervised training over a set of labeled digital images corresponding to a plurality of base classes, wherein modifying the decision boundaries facilitates classification into one or more additional classes not present in the plurality of base classes;
and
determine a classification for the digital image by processing the extracted features using a classification neural network comprising a classifier trained in connection with the feature extractor to classify the digital image into an additional class of the one or more additional classes not present in the plurality of base classes.

A system comprising:

one or more memory devices comprising a set of labeled digital images corresponding to a plurality of base classes, a base neural network comprising a feature extractor and a first classifier, and a classification neural network comprising the feature extractor and a second classifier;
and
one or more server devices that cause the system to:
train, based on the set of labeled digital images, the base neural network comprising the feature extractor and the first classifier to classify a digital image into one of the plurality of base classes by:
determining a self-supervised loss associated with the feature extractor utilizing a self-supervised training process;
generating, utilizing the feature extractor based on the self-supervised loss, a feature manifold comprising features associated with the plurality of base classes;
and
modifying, utilizing manifold mixup regularization, the feature manifold based on a manifold mixup loss associated with the feature extractor;
and
train the classification neural network comprising the feature extractor and the second classifier to classify the digital image into an additional class not present in the plurality of base classes based on the modified feature manifold.

7 more elements

Dependencies are inferred from the claim text.

Facts

Application

16/661,617Filed 23 Oct 2019

Published

US 2021/0124993 A129 Apr 2021

Granted

19 Apr 2022

Term ends

Not estimatedThe source records don’t support an estimate.

Classification18 codes

G06N 3/08
Neural networks: learning methods
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/2415
Relating to the classification model, e.g. parametric or non-parametric approaches: based on parametric or probabilistic models, e.g. based on…
G06F 18/2431
Relating to the number of classes: multiple classes

Classifying digital images in few-shot tasks based on neural networks trained using manifold mixup regularization and self-supervision

Life of this patent

Classifying digital images in few-shot tasks based on neural networks trained using manifold mixup regularization and self-supervision

Life of this patent

Abstract

Claims20