Progressive neural networks

A method for sequentially training a sequence of deep neural networks (DNNs) to perform a sequence of machine learning tasks, each DNN in the sequence corresponding to a respective machine learning task, and the method comprising:

• for a first machine learning task in the sequence:
• training a first DNN in the sequence that corresponds to the first machine learning task to perform the first machine learning task, wherein (i) the first DNN comprises a first plurality of indexed layers, and (ii) each layer in the first plurality of indexed layers is configured to receive a respective layer input and process the layer input to generate a respective layer output;
• for each subsequent machine learning task in the sequence:
• training a subsequent DNN corresponding to the machine learning task to perform the machine learning task, wherein (i) the subsequent DNN comprises a subsequent plurality of indexed layers, and (ii) each layer with index greater than one in the subsequent indexed plurality of layers receives input from (i) a preceding layer of the subsequent DNN, and (ii) one or more preceding layers of respective preceding DNNs, wherein a preceding layer is a layer whose index is one less than a current index, and wherein the input from one or more preceding layers of respective preceding DNNs is generated by applying respective non-linear lateral connections to the respective layer outputs of the one or more preceding layers of respective preceding DNNs the respective non-linear lateral connections representing a learned, non-linear transformation of the respective layer outputs of the one or more preceding layers of the respective preceding DNNs.

A method for processing an input using a sequence of deep neural networks (DNNs), wherein each DNN in the sequence of DNNs has been trained to perform a respective machine learning task, the sequence of DNN comprising:

• a first DNN that corresponds to a first machine learning task, wherein (i) the first DNN comprises a first plurality of indexed layers, and (ii) each layer in the first plurality of indexed layers is configured to receive a respective layer input and process the layer input to generate a respective layer output;
• and
• one or more subsequent DNNs corresponding to one or more respective machine learning tasks, wherein (i) each subsequent DNN comprises a respective plurality of indexed layers, and (ii) each layer in a respective plurality of indexed layers with index greater than one receives input from (i) a preceding layer of the respective subsequent DNN, and (ii) one or more preceding layers of respective preceding DNNs, wherein a preceding layer is a layer whose index is one less than a current index, and wherein the input from one or more preceding layers of respective preceding DNNs is generated by applying respective non-linear lateral connections to the respective layer outputs of the one or more preceding layers of respective preceding DNNs wherein the respective non-linear lateral connections represent a learned, non-linear transformation of the respective layer outputs of the one or more preceding layers of the respective preceding DNNs;
• the method comprising:
• receiving an input as part of a machine learning task corresponding to a last subsequent DNN in the sequence of DNNs;
• and
• processing the input using the last subsequent DNN in the sequence to generate a last subsequent DNN output for the machine learning task.