Connection weight learning for guided architecture evolution

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for determining a neural network architecture of a neural network for performing a video processing neural network task, the operations comprising:

• maintaining data defining a set of neural network architectures, wherein for each neural network architecture:
• the neural network architecture comprises a plurality of blocks, wherein each block is a space-time convolutional block comprising one or more neural network layers that is configured to process a block input to generate a block output and
• for each of one or more given blocks:
• (i) the block input for the given block comprises a block output from each of a plurality of other blocks, (ii) the given block has a respective connection weight parameter corresponding to each of the plurality of other blocks, and (iii) processing the block input comprises combining the other block outputs using the connection weight parameters corresponding to the other blocks;
• at each of a plurality of iterations:
• selecting a parent neural network architecture from the set of neural network architectures;
• training a neural network having the parent neural network architecture to perform the video processing neural network task, comprising determining trained values of the connection weight parameters of the parent neural network architecture;
• generating a new neural network architecture based at least in part on the trained values of the connection weight parameters of the parent neural network architecture;
• and
• adding the new neural network architecture to the set of neural network architectures;
• and
• after a final iteration of the plurality of iterations, selecting a final neural network architecture from the set of neural network architectures based on a performance metric for the final neural network architecture on the video processing neural network task.

One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for determining a neural network architecture of a neural network for performing a video processing neural network task, the operations comprising:

• maintaining data defining a set of neural network architectures, wherein for each neural network architecture:
• the neural network architecture comprises a plurality of blocks, wherein each block is a space-time convolutional block comprising one or more neural network layers that is configured to process a block input to generate a block output and
• for each of one or more given blocks:
• (i) the block input for the given block comprises a block output from each of a plurality of other blocks, (ii) the given block has a respective connection weight parameter corresponding to each of the plurality of other blocks, and (iii) processing the block input comprises combining the other block outputs using the connection weight parameters corresponding to the other blocks;
• at each of a plurality of iterations:
• selecting a parent neural network architecture from the set of neural network architectures;
• training a neural network having the parent neural network architecture to perform the video processing neural network task, comprising determining trained values of the connection weight parameters of the parent neural network architecture;
• generating a new neural network architecture based at least in part on the trained values of the connection weight parameters of the parent neural network architecture;
• and
• adding the new neural network architecture to the set of neural network architectures;
• and
• after a final iteration of the plurality of iterations, selecting a final neural network architecture from the set of neural network architectures based on a performance metric for the final neural network architecture on the video processing neural network task.