US 11,599,794 B1Granted

System and method for training sample generator with few-shot learning

AssigneeMoffett International Co., Limited

InventorEnxu Yan

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for few-shot learning-based generator training are disclosed. An exemplary method may start with obtaining a teacher model and a plurality of training samples, as well as a generator for generating more training samples. After generating a plurality of additional training samples using the method may continue with feeding the plurality of generated additional training samples into the teacher model to obtain a plurality of first statistics; and feeding the plurality of training samples into the teacher model to obtain a plurality second statistics. Then the method further includes training the generator to minimize a distance between the plurality of first statistics and the plurality of second statistics.

Claims17

Dependent claims

A computer-implemented method for training a generator using few shot training by collecting statistics inside teacher models, and the method comprising:

obtaining a teacher model and a plurality of training samples;
generating a plurality of samples using the generator;
feeding the plurality of generated samples into the teacher model to obtain a plurality of first statistics, wherein the plurality of first statistics are determined based on outputs of more than one layers in the teacher model when the plurality of generated samples are passing through the teacher model, wherein the outputs comprise one or more tensors generated by each of the more than one layers, and the plurality of first statistics are obtained by:
for each of the one or more tensors, determining one or more channel-level statistics, wherein the one or more channel-level statistics comprise a channel mean, a channel variance, and an i-th order moment of all values within a corresponding channel in the tensor, i is greater than two;
feeding the plurality of obtained training samples into the teacher model to obtain a plurality of second statistics, wherein the plurality of second statistics are determined based on outputs of the more than one layers in the teacher model when the plurality of obtained training samples are passing through the teacher model;
training the generator by minimizing a distance between the plurality of first statistics and the plurality of second statistics;
and
generating new training samples using the trained generator for knowledge distillation from the teacher model to a student model, wherein the new training samples resemble the plurality of obtained training samples with regard to features related to an intended application of the student model, and are allowed to deviate from the plurality of obtained training samples with regard to features irrelevant to the intended application of the student model.

5 more elements

A system for training a generator using few shot training by collecting statistics inside teacher models, the system comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising:

obtaining a teacher model and a plurality of training samples;
generating a plurality of samples using the generator;
feeding the plurality of generated samples into the teacher model to obtain a plurality of first statistics, wherein the plurality of first statistics are determined based on outputs of more than one layers in the teacher model when the plurality of generated samples are passing through the teacher model, wherein the outputs comprise one or more tensors generated by each of the more than one layers, and the obtaining of the plurality of first statistics comprises:
for each of the one or more tensors, determining one or more channel-level statistics, wherein the one or more channel-level statistics comprise a channel mean, a channel variance, and an i-th order moment of all values within a corresponding channel in the tensor, i is greater than two;
feeding the plurality of obtained training samples into the teacher model to obtain a plurality of second statistics, wherein the plurality of second statistics are determined based on outputs of the more than one layers in the teacher model when the plurality of obtained training samples are passing through the teacher model;
training the generator by minimizing a distance between the plurality of first statistics and the plurality of second statistics;
and
generating new training samples using the trained generator for knowledge distillation from the teacher model to a student model, wherein the new training samples resemble the plurality of obtained training samples with regard to features related to an intended application of the student model, and are allowed to deviate from the plurality of obtained training samples with regard to features irrelevant to the intended application of the student model.

5 more elements

A non-transitory computer-readable storage medium for training a generator using few shot training by collecting statistics inside teacher models, the storage medium being configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

obtaining a teacher model and a plurality of training samples;
generating a plurality of samples using the generator;
feeding the plurality of generated samples into the teacher model to obtain a plurality of first statistics, wherein the plurality of first statistics are determined based on outputs of more than one layers in the teacher model when the plurality of generated samples are passing through the teacher model, wherein the outputs comprise one or more tensors generated by each of the more than one layers, and the obtaining of the plurality of first statistics comprises:
for each of the one or more tensors, determining one or more channel-level statistics, wherein the one or more channel-level statistics comprise a channel mean, a channel variance, and an i-th order moment of all values within a corresponding channel in the tensor, i is greater than two;
feeding the plurality of obtained training samples into the teacher model to obtain a plurality of second statistics, wherein the plurality of second statistics are determined based on outputs of the more than one layers in the teacher model when the plurality of obtained training samples are passing through the teacher model;
training the generator by minimizing a distance between the plurality of first statistics and the plurality of second statistics;
and
generating new training samples using the trained generator for knowledge distillation from the teacher model to a student model, wherein the new training samples resemble the plurality of obtained training samples with regard to features related to an intended application of the student model, and are allowed to deviate from the plurality of obtained training samples with regard to features irrelevant to the intended application of the student model.

5 more elements

Dependencies are inferred from the claim text.