US 12,260,336 B2Granted

Training a neural network

AssigneeImagination Technologies Limited

InventorsMuhammad Asad, Elia Condorelli, Cagatay Dikici

Life of this patent

Estimating the term…

Loading patent…

Abstract

A computer implemented method of training a neural network configured to combine a set of coefficients with respective input data values. So as to train a test implementation of the neural network, sparsity is applied to one or more of the coefficients according to a sparsity parameter, the sparsity parameter indicating a level of sparsity to be applied to the set of coefficients; the test implementation of the neural network is operated on training input data using the coefficients so as to form training output data; in dependence on the training output data, assessing the accuracy of the neural network; the sparsity parameter is updated in dependence on the accuracy of the neural network; and a runtime implementation of the neural network is configured in dependence on the updated sparsity parameter.

Claims20

Dependent claims

A computer implemented method of training a neural network configured to combine a set of coefficients with respective input data values, the method comprising:

so as to train a test implementation of the neural network:
applying sparsity,
wherein applying sparsity comprises:
dividing the set of coefficients into multiple groups of coefficients, wherein
each group comprises a plurality of coefficients of the set of coefficients;
representing each group of coefficients of the multiple groups of coefficients by a respective single value;
modelling those values using a differentiable function representative of an extreme value distribution so as to identify a threshold value in dependence on a sparsity parameter, wherein the sparsity parameter indicates a level of sparsity to be applied to the set of coefficients;
and
applying sparsity to a plurality of groups of coefficients of the multiple groups of coefficients in dependence on that threshold value, wherein applying sparsity to each group of coefficients of the plurality of groups of coefficients comprises setting each of the coefficients in that group to zero;
operating the test implementation of the neural network on training input data using the coefficients so as to form training output data;
in dependence on the training output data, assessing the accuracy of the neural network;
and
updating the sparsity parameter in dependence on the accuracy of the neural network by modifying the threshold value by backpropagating one or more gradient vectors using the differentiable function;
and
configuring a runtime implementation of the neural network in dependence on the updated sparsity parameter that, when implemented at a data processing system, executes the runtime implementation of neural network in dependence on the updated sparsity parameter.

12 more elements

A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a computer implemented method of training a neural network configured to combine a set of coefficients with respective input data values, the method comprising:

so as to train a test implementation of the neural network:
applying sparsity,
wherein applying sparsity comprises:
dividing the set of coefficients into multiple groups of coefficients, wherein each group comprises a plurality of coefficients of the set of coefficients;
representing each group of coefficients of the multiple groups of coefficients by a respective single value;
modelling those values using a differentiable function representative of an extreme value distribution so as to identify a threshold value in dependence on a sparsity parameter, wherein the sparsity parameter indicates a level of sparsity to be applied to the set of coefficients;
and
applying sparsity to a plurality of groups of coefficients of the multiple groups of coefficients in dependence on that threshold value, wherein applying sparsity to each group of coefficients of the plurality of groups of coefficients comprises setting each of the coefficients in that group to zero;
operating the test implementation of the neural network on training input data using the coefficients so as to form training output data;
in dependence on the training output data, assessing the accuracy of the neural network;
and
updating the sparsity parameter in dependence on the accuracy of the neural network by modifying the threshold value by backpropagating one or more gradient vectors using the differentiable function;
and
configuring a runtime implementation of the neural network in dependence on the updated sparsity parameter that, when implemented at a data processing system, executes the runtime implementation of neural network in dependence on the updated sparsity parameter.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/559,331Filed 22 Dec 2021
Priority: GB 202038622 Dec 2020
Published: US 2022/0261652 A118 Aug 2022
Granted: 25 Mar 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G06N 3/084
Learning methods: backpropagation, e.g. using gradient descent
G06F 18/2136
Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods: based on sparsity criteria, e.g. with…
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks
G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]