US 11,775,833 B2 · Accelerated TR-L-BFGS algorithm for neural network

Abstract

Techniques herein train a multilayer perceptron, sparsify edges of a graph such as the perceptron, and store edges and vertices of the graph. Each edge has weight. A computer sparsifies perceptron edges. The computer performs a forward-backward pass on the perceptron to calculate a sparse Hessian matrix. Based on that Hessian, the computer performs quasi-Newton perceptron optimization. The computer repeats this until convergence. The computer stores edges in an array and vertices in another array. Each edge has weight and input and output indices. Each vertex has input and output indices. The computer inserts each edge into an input linked list based on its weight. Each link of the input linked list has the next input index of an edge. The computer inserts each edge into an output linked list based on its weight. Each link of the output linked list comprises the next output index of an edge.

Claims20

Dependent claims

A method comprising:

storing an artificial neural network that comprises a plurality of edges and a plurality of vertices, wherein:
each edge of the plurality of edges is associated with a respective weight, and
each edge of the plurality of edges originates at a respective source vertex of the plurality of vertices and terminates at a respective target vertex of the plurality of vertices;
training the artificial neural network;
and
repeatedly during said training:
a) calculating a statistic of weights of a sampled subset of edges in the artificial neural network and, based on the statistic of the weights of the sampled subset of edges, a threshold that maintains a utility of each vertex in the plurality of vertices;
and
b) removing, when a sum of weights of edges that originate or terminate at the source vertex or the target vertex of a particular edge of the plurality of edges exceeds said threshold, the particular edge from the artificial neural network;
wherein the method is performed by one or more computers.

7 more elements

Dependencies are inferred from the claim text.

Accelerated TR-L-BFGS algorithm for neural network

Life of this patent

Accelerated TR-L-BFGS algorithm for neural network

Life of this patent

Abstract

Claims20