Circulant neural networks

A method for training a neural network that includes a plurality of neural network layers on a plurality of training data inputs, wherein the plurality of neural network layers includes a circulant neural network layer that is associated with a parameter matrix that is a circulant matrix, and wherein the method comprises, for each plurality of training data inputs and for the circulant layer:

• receiving, by one or more computers, the training data input for the circulant layer;
• processing, by the one or more computers, the training data input to generate a layer output for the circulant layer, comprising:
• identifying the parameter matrix for the circulant layer, wherein the parameter matrix is a circulant matrix that is fully specified by a single vector;
• determining a product between the parameter matrix and the training data input to the circulant layer, comprising performing a circular convolution between the single vector that fully specifies the parameter matrix and the training data input using a Fast Fourier Transform (FFT) in place of a multiplication between the parameter matrix and the training data input, and
• computing an activation function that is dependent on the product of the parameter matrix and the training data input to generate a layer output for the circulant layer;
• receiving, by the one or more computers, a back-propagated gradient for the training data input from the neural network layer above the circulant layer;
• computing, by the one or more computers, the gradient of an error function for the circulant layer, wherein the error function is dependent on the product of the circulant matrix associated with the circulant layer and the received back-propagated gradient, and wherein computing the gradient of the error function comprises converting circulant matrix multiplication to circular convolution and performing a Fast Fourier Transform (FFT);
• and
• updating, by the one or more computers, the entries of the circulant matrix associated with the circulant layer using the computed gradient.

A neural network system implemented by one or more computers, the neural network system comprising:

• a circulant neural network layer, wherein the circulant layer is associated with a parameter matrix that is a circulant matrix, and wherein the circulant neural network layer is configured to, during processing of an input to the neural network system to generate an output from the input, perform operations comprising:
• receiving a layer input for the circulant layer;
• identifying the parameter matrix for the circulant layer, wherein the parameter matrix is fully specified by a single vector;
• determining a product between the parameter matrix and the layer input to the circulant layer, comprising performing a circular convolution between the single vector that fully specifies the parameter matrix and the layer input using a Fast Fourier Transform (FFT) in place of a multiplication between the parameter matrix and the layer input;
• and
• computing an activation function that is dependent on the product of the parameter matrix and the layer input to generate a layer output for the circulant layer.