Back to results
US 10,474,925 B2Granted
Deep neural network with side branches for recognizing and classifying media data and method for using the same
AssigneeIndustrial Technology Research Institute
InventorsMao-Yu Huang, Ching-Hao Lai
Life of this patent
Estimating the term…
Abstract
A deep neural network and a method for recognizing and classifying a multimedia data as one of a plurality of pre-determined data classes with enhanced recognition and classification accuracy and efficiency are provided. The use of the side branch(es) (or sub-side branch(es), sub-sub-side branch(es), and so on) extending from the main branch (or side branch(es), sub-side branch(es), and so on), the sequential decision making mechanism, and the collaborating (fusing) decision making mechanism in a deep neural network would equip a deep neural network with the capability for fast forward inference so as to enhance recognition and classification accuracy and efficiency of the deep neural network.
Claims42
A system for recognizing and classifying a media data as one of a plurality of pre-determined data classes using a deep neural network, comprising:
- a non-transitory computer-readable medium containing computer-executable instructions for recognizing and classifying the media data as the one of a plurality of pre-determined data classes using the deep neural network;
- and
- a processor configured to execute the computer-executable instructions,
- wherein the deep neural network comprises:
- a main path in a sequential order of an input layer for receiving a media data, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups of layers in the main path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the main path, wherein X>1 and X is an integer;
- at least one alternative path in a sequential order of the input layer, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups in the at least one alternative path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the at least one alternative path, wherein the X groups of layers in each of the at least one alternative path consist of respective Y groups of the X groups of layers in the main path, and additional X-Y groups of layers as a side branch extending from a respective Yth group of the X groups in the main path, wherein the respective Y groups of the X groups of layers in the main path are a first group of the X groups of layers in the main path through the respective Yth group of the X groups of layers in the main path and wherein the respective Y for each of the at least one alternative path is an integer and 1≤Y<X;
- a fusion layer, wherein the classification layer of the main path and the classification layer of the at least one alternative path merge at the fusion layer;
- and
- a predictor configured to recognize and classify the media data as one of the plurality of pre-determined data classes corresponding to a final class likelihood,
- wherein the deep neural network directs the media data sequentially through one or more of the at least one alternative path and the main path one at a time until the final class likelihood is outputted, and outputs either a highest class likelihood of a first-ever one of the at least one alternative path and the main path that reaches or exceeds a corresponding class likelihood threshold or a highest fused class likelihood from the fusion layer based on the class likelihoods of two or more of any of the main path and the at least one alternative path that the media data has been directed through as the final class likelihood.
7 more elements
A method for recognizing and classifying a media data as one of a plurality of pre-determined data classes using a deep neural network,
- the deep neural network comprising:
- a main path in a sequential order of an input layer for receiving a media data, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups of layers in the main path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the main path, wherein X>1 and X is an integer;
- at least one alternative path in a sequential order of the input layer, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups in the at least one alternative path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the at least one alternative path, wherein the X groups of layers in each of the at least one alternative path consist of respective Y groups of the X groups of layers in the main path, and additional X-Y groups of layers as a side branch extending from a respective Yth group of the X groups in the main path, wherein the respective Y groups of the X groups of layers in the main path are a first group of the X groups of layers in the main path through the respective Yth group of the X groups of layers in the main path and wherein the respective Y for each of the at least one alternative path is an integer and 1≤Y<X;
- a fusion layer, wherein the classification layer of the main path and the classification layer of the at least one alternative path merge at the fusion layer;
- and
- a predictor configured to recognize and classify the media data as one of the plurality of pre-determined data classes corresponding to a final class likelihood,
- the method comprising:
- directing the media data sequentially through one or more of the at least one alternative path and the main path one at a time until the final class likelihood is outputted;
- outputting either a highest class likelihood of a first-ever one of the at least one alternative path and main path that reaches or exceeds a corresponding class likelihood threshold or a highest fused class likelihood from the fusion layer based on the class likelihoods of two or more of any of the main path and the at least one alternative path that the media data has been directed through as the final class likelihood;
- and
- recognizing and classifying the media data as one of the plurality of pre-determined data classes corresponding to the final class likelihood.
8 more elements
A non-transitory computer-readable medium containing a computer program product comprising computer-executable instructions for recognizing and classifying a media data as one of a plurality of pre-determined data classes using a deep neural network,
- the deep neural network comprising:
- a main path in a sequential order of an input layer for receiving a media data, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups of layers in the main path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the main path, wherein X>1 and X is an integer;
- at least one alternative path in a sequential order of the input layer, only X groups of layers for extracting features from the media data, at least one pooling layer for downsampling an output from the X groups in the at least one alternative path, and a classification layer for computing a class likelihood for each of the plurality of pre-determined data classes for the media data through the at least one alternative path, wherein the X groups of layers in each of the at least one alternative path consist of respective Y groups of the X groups of layers in the main path, and additional X-Y groups of layers as a side branch extending from a respective Yth group of the X groups in the main path, wherein the respective Y groups of the X groups of layers in the main path are a first group of the X groups of layers in the main path through the respective Yth group of the X groups of layers in the main path and wherein the respective Y for each of the at least one alternative path is an integer and 1≤Y<X;
- a fusion layer, wherein the classification layer of the main path and the classification layer of the at least one alternative path merge at the fusion layer;
- and
- a predictor configured to recognize and classify the media data as one of the plurality of pre-determined data classes corresponding to a final class likelihood,
- the computer program product comprising the computer-executable instructions for:
- directing the media data sequentially through one or more of the at least one alternative path and the main path one at a time until the final class likelihood is outputted;
- outputting either a highest class likelihood of a first-ever one of the at least one alternative path and main path that reaches or exceeds a corresponding class likelihood threshold or a highest fused class likelihood from the fusion layer based on the class likelihoods of two or more of any of the main path and the at least one alternative path that the media data has been directed through as the final class likelihood;
- and
- recognizing and classifying the media data as one of the plurality of pre-determined data classes corresponding to the final class likelihood.
8 more elements
Dependencies are inferred from the claim text.