Generating larger neural networks

A method performed by one or more computers, the method comprising:

• identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value;
• generating a larger neural network having a larger neural network structure, the larger neural network structure comprising:
• (i) the plurality of original neural network units, and
• (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters;
• initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units;
• and
• training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value;
• generating a larger neural network having a larger neural network structure, the larger neural network structure comprising:
• (i) the plurality of original neural network units, and
• (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters;
• initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units;
• and
• training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value;
• generating a larger neural network having a larger neural network structure, the larger neural network structure comprising:
• (i) the plurality of original neural network units, and
• (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters;
• initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units;
• and
• training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values.