# US 2019/0294967 A1 · Circulant neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294967 A1
- Document ID: US20190294967A1
- Status: Published application
- Kind: A1
- Application number: 15/014,804
- Filed: 2016-02-03
- Published: 2019-09-26
- Earliest claimed: 2015-02-03
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Sanjiv Kumar; Xinnan Yu
- Web page: https://patentgrep.com/patent/US20190294967A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using a neural network that includes a circulant neural network layer. One of the methods includes receiving a layer input for the circulant layer; and processing the layer input to generate a layer output for the circulant layer, wherein processing the layer input comprises computing an activation function, wherein the activation function is dependent on the product of the circulant matrix associated with the circulant layer and the layer input, and wherein computing the activation function comprises performing a circular convolution using a Fast Fourier Transform (FFT).

## Claims

### Claim 1

A method for processing an input through each of a plurality of layers of a neural network to generate an output, wherein the plurality of layers comprises a circulant layer that is associated with a parameter matrix that is a circulant matrix, and wherein the method comprises:
receiving, by one or more computers, a layer input for the circulant layer;
identifying, by the one or more computers, the parameter matrix for the circulant layer, wherein the parameter matrix is fully specified by a single vector;
determining, by the one or more computers, a product between the parameter matrix and the layer input to the circulant layer, comprising performing a circular convolution between the single vector that fully specifies the parameter matrix and the layer input using a Fast Fourier Transform (FFT) in place of a multiplication between the parameter matrix and the layer input; and
computing, by the one or more computers, an activation function that is dependent on the product of the parameter matrix and the layer input to generate a layer output for the circulant layer.

### Claim 2 (depends on 1)

The method of claim 1, wherein the single vector appears as a first row of the matrix, and wherein each subsequent row vector in the circulant matrix is a vector whose entries are rotated one entry to the right relative to a preceding row vector in the circulant matrix.

### Claim 3 (depends on 1)

The method of claim 1, wherein the single vector appears as a first column of the matrix, and wherein each subsequent column in the circulant matrix is a column vector whose entries are rotated one entry below relative to a preceding column in the circulant matrix.

### Claim 4

(canceled)

### Claim 5 (depends on 1)

The method of claim 1, wherein the circulant layer receives the layer input for the circulant layer from a first layer having a first number of nodes and provides the layer output to a second neural network layer having a second number of nodes, wherein the layer input is a vector with dimension equal to the first number of nodes, and wherein the layer output is a vector with dimension equal to the second number of nodes.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first number equals the second number, and wherein the layer output is a vector produced by computing the activation function.

### Claim 7 (depends on 5)

The method of claim 5, wherein the first number is greater than the second number, and wherein the layer output is the first k elements of a vector produced by computing the activation function as the layer output, wherein k is equal to the second number.

### Claim 8 (depends on 5)

The method of claim 5, wherein the first number is less than the second number, and wherein computing the activation function to generate the layer output for the circulant layer comprises padding k−d predetermined constant values on the end of the vector produced by computing the activation function, wherein k is equal to the second number and d is equal to the first number.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining a product between the parameter matrix and the layer input to the circulant layer further comprises performing a random sign flipping on the layer input prior to determining the product.

### Claim 10 (depends on 9)

The method of claim 9, wherein performing the random sign flipping on the layer input comprises applying a diagonal matrix to the layer input, wherein the diagonal matrix is a matrix whose entries outside the main diagonal are all zero, and the diagonal entries on the main diagonal are Bernoulli random variables, wherein the Bernoulli random variables take the values +1 and −1 with equal probability.

### Claim 11 (depends on 1)

The method of claim 1, wherein each entry of the single vector is generated independently from a standard normal distribution.

### Claim at position 12



### Claim 13

A method for training a neural network that includes a plurality of neural network layers on a plurality of training data inputs, wherein the plurality of neural network layers includes a circulant neural network layer that is associated with a parameter matrix that is a circulant matrix, and wherein the method comprises, for each plurality of training data inputs and for the circulant layer:
receiving, by one or more computers, the training data input for the circulant layer;
processing, by the one or more computers, the training data input to generate a layer output for the circulant layer, comprising:
identifying the parameter matrix for the circulant layer, wherein the parameter matrix is a circulant matrix that is fully specified by a single vector;
determining a product between the parameter matrix and the training data input to the circulant layer, comprising performing a circular convolution between the single vector that fully specifies the parameter matrix and the training data input using a Fast Fourier Transform (FFT) in place of a multiplication between the parameter matrix and the training data input, and
computing an activation function that is dependent on the product of the parameter matrix and the training data input to generate a layer output for the circulant layer;
receiving, by the one or more computers, a back-propagated gradient for the training data input from the neural network layer above the circulant layer;
computing, by the one or more computers, the gradient of an error function for the circulant layer, wherein the error function is dependent on the product of the circulant matrix associated with the circulant layer and the received back-propagated gradient, and wherein computing the gradient of the error function comprises converting circulant matrix multiplication to circular convolution and performing a Fast Fourier Transform (FFT); and
updating, by the one or more computers, the entries of the circulant matrix associated with the circulant layer using the computed gradient.

### Claim 14 (depends on 13)

The method of claim 13 wherein the single vector appears as a first row of the matrix, and wherein each subsequent row vector in the circulant matrix is a vector whose entries are rotated one entry to the right relative to a preceding row vector in the circulant matrix.

### Claim 15 (depends on 13)

The method of claim 13, wherein the single vector appears as a first column of the matrix, and wherein each subsequent column in the circulant matrix is a column vector whose entries are rotated one entry below relative to a preceding column in the circulant matrix.

### Claim 16 (depends on 13)

The method of claim 13, wherein converting circulant matrix multiplication to circular convolution comprises replacing the product of the circulant matrix associated with the circulant layer and the training data input with the circular convolution of the single vector that fully specifies the circulant matrix and the training data input.

### Claim 17

A neural network system implemented by one or more computers, the neural network system comprising:
a circulant neural network layer, wherein the circulant layer is associated with a parameter matrix that is a circulant matrix, and wherein the circulant neural network layer is configured to, during processing of an input to the neural network system to generate an output from the input, perform operations comprising:
receiving a layer input for the circulant layer;
identifying the parameter matrix for the circulant layer, wherein the parameter matrix is fully specified by a single vector;
determining a product between the parameter matrix and the layer input to the circulant layer, comprising performing a circular convolution between the single vector that fully specifies the parameter matrix and the layer input using a Fast Fourier Transform (FFT) in place of a multiplication between the parameter matrix and the layer input; and
computing an activation function that is dependent on the product of the parameter matrix and the layer input to generate a layer output for the circulant layer.

### Claim 18 (depends on 17)

The neural network system of claim 17, wherein the single vector appears as a first row of the matrix, and wherein each subsequent row vector in the circulant matrix is a vector whose entries are rotated one entry to the right relative to a preceding row vector in the circulant matrix.

### Claim 19 (depends on 17)

The neural network system of claim 17, wherein the single vector appears as a first column of the matrix, and wherein each subsequent column in the circulant matrix is a column vector whose entries are rotated one entry below relative to a preceding column in the circulant matrix.

### Claim 20

(canceled)
