Block-diagonal hessian-free optimization for recurrent and convolutional neural networks

A method for training a neural network, the method comprising:

• identifying, using a block diagonal Hessian free optimizer stored in memory of a computing device, a first block and a second block from non-overlapping layers the neural network, wherein the first block has first adjacent layers of the layers and the second block has second adjacent layers of the layers;
• generating a gradient from a gradient mini-batch included in training data;
• generating a block diagonal Hessian matrix of curvature-vector products from curvature mini-batches included in the training data, wherein data in the gradient mini-batch is different from data in the curvature mini-batch and wherein the block diagonal Hessian matrix corresponds to parameter pairs in the neural network, first pairs in the pairs including at least one first parameter of the first block and at least one second parameter of the second block, second pairs in the pairs including two of the at least one first parameter, and third pairs in the pairs including two of the at least one second parameter;
• setting curvature-vector products in the block diagonal Hessian matrix to zero values for one or more terms that correspond to pairs that include parameters from different blocks;
• generating a first conjugate gradient for the first block from the gradient and the curvature-vector products in the block diagonal Hessian matrix and a second conjugate gradient for the second block in parallel with the first conjugate gradient;
• determining, using the first conjugate gradient a change in the at least one first parameter of the first block;
• and
• determining, using the second conjugate gradient a change in the at least one second parameter of the second block.

A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations that train a neural network, the operations comprising:

• identifying, using a block diagonal Hessian free optimizer stored in memory of a computing device, a first block and a second block from layers in the neural network, wherein the first block has first adjacent layers of the layers of the neural network and the second block has second adjacent layers of the layers of the neural network;
• generating a gradient from a gradient mini-batch included in training data;
• generating a curvature-vector product from a curvature mini-batch included in the training data;
• generating a conjugate gradient from the gradient and the curvature-vector product;
• determining, using the conjugate gradient a change in at least one first parameter of the first block;
• determining, using the conjugate gradient a change in at least one second parameter of the second block;
• and
• determining the at least one first parameter in the first block using the at least one first parameter and the change in the at least one first parameter, and the at least one second parameter in the second block using the at least one second parameter and the change in the at least one second parameter.