Loading patent…
AssigneeCanon Kabushiki Kaisha
InventorsJunjie Liu, Tsewei Chen, Dongchao Wen, Hongxing Gao, Wei Tao
Estimating the term…
A training and application method for a neural network model is provided. The training method determines the first network model to be trained and sets a downscaling layer for at least one layer in the first network model, wherein the number of filters and filter kernel of the downscaling layer are identical to those of layers to be trained in the second network model. Filter parameters of the downscaling layer are transmitted to the second network model as training information. By this training method, training can also be performed even when the scale of the layer for training in the first network model is different from that of the layers to be trained in the second network model, and the amount of lost data is small.
A method of training a multi-layer neural network model, comprising:
A system for training a multi-layer neural network model, comprising:
An apparatus for training a multi-layer neural network model, comprising:
An application method of a multi-layer neural network model comprising:
An application apparatus of a multi-layer neural network model, comprising:
A non-transitory computer-readable storage medium storing instructions for causing a computer to perform a training method of a multi-layer neural network model according to a training method when executed by the computer, the training method comprises:
Dependencies are inferred from the claim text.
Drawings are temporarily unavailable. Open the PDF at USPTO
From the US filing history. Foreign family members aren’t listed.
Learning methods: non-supervised learning, e.g. competitive learning
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: validation; Performance…
Architecture, e.g. interconnection topology: combinations of networks
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
About 13 years of term left · ends 31 Oct 2039 (estimated)
This application claims priority from Chinese Patent Application Number 201811331677.8 file on Nov. 9, 2018, the entirety of the disclosure being incorporated herein by reference.
The present disclosure relates to a modeling field of Deep Neural Networks (DNN), in particular to a method of training other DNN models by using the trained DNN model.
DNN model is a network model that has been successfully applied in the image processing field, especially in the fields of image classification, target detection, image division or the like. However, since a DNN model usually has a complicated network architecture and contains a large number of model parameters, a DNN model has the disadvantages of large computation and slow processing speed. This makes it difficult for DNN to be applied to real-time services, especially real-time services of embedded devices with poor computing capability.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments of the present disclosure and, together with the description of the exemplary embodiments, serve to explain the principles of the present disclosure.
FIG. 1 is a schematic diagram of KD technique.
FIG. 2A and FIG. 2B illustrate examples where scales are identical and different, respectively.
The KD technique is a technique which trains a DNN model with a simple architecture by using feature map and output extracted from a DNN model in which the training has been completed and has a complicated network architecture, so as to make output of the DNN model with the simple architecture and output of the DNN model with the complicated network architecture being approximate. FIG. 1 illustrates a schematic diagram of KD technique by taking a Convolutional Neural Network (CNN) model as an example.
Here, a network model in which the training has been completed, has a complicated network architecture and a strong data processing capability is called a Teacher network (abbreviated as T network) model, which is used to provide a network model to be trained with information for training, such as feature map information or other information. A network model to be trained with a simple network architecture and a relatively weak data processing capability is called a Student network (abbreviated as S network) model. S network model performs an optimization training on its own network model based on information provided by T network model until the output of S network model is approximate to that of T network model.
FIG. 1 shows an example of training a convolutional layer in S network model by extracting feature map from a convolutional layer of T network model. At every time of training, a computation is performed layer by layer from top to bottom in the S network model based on input of the previous layer, output (feature map) currently to be trained in the S network model is compared with feature map for training provided by the T network model, and model parameters in the S network model is optimized according to the comparison results, such that residual error between the feature map to be trained in the S network model and the feature map for training provided by the T network model become small gradually. After several times of training, the training of the S network model is completed when the residual error between the output of the S network model and the output of the T network model is less than the set value. Since the S network model has the characteristics that network architecture is simple and the contained model parameters are less, if the S network model of which training has been completed is used to replace the application of the T network model in the real-time services, especially in the real-time services of embedded devices, the S network model also has such advantages that the occupied storage space is small, calculation amount required in operation is less and inference time is short in a case where the application result and the T network model are approximate.