Back to results
US 12,682,240 B1Granted
Variance aware techniques to prune neural networks
AssigneeNVIDIA Corporation
InventorsShuang Gao, Xin Liu, John Zedlewski
Life of this patent
Estimating the term…
Abstract
Systems and methods to use weight information corresponding to a neural network, wherein weight information of at least a first one or more layers of the neural network is modified to compensate for deactivating one or more second layers of the neural network. The neural network may be a residual neural network.
Claims46
One or more processors, comprising:
- circuitry to:
- identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;
- perform a regularization on the one or more filters;
- group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
5 more elements
A system, comprising:
- one or more computers comprising one or more processors to:
- identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;
- perform a regularization on the one or more filters;
- group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
5 more elements
A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
- identify one or more filters of one or more first layers of a neural network that are to be removed from each of the one or more first layers;
- perform a regularization on the one or more filters;
- group a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- align structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determine, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivate one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
4 more elements
One or more processors, comprising:
- circuitry to help train a neural network by at least:
- identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;
- performing a regularization on the one or more filters;
- grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
5 more elements
A system, comprising:
- one or more computers comprising one or more processors to help train a neural network by at least:
- identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;
- performing a regularization on the one or more filters;
- grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
5 more elements
A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
- help train a neural network by:
- identifying one or more filters of one or more first layers of the neural network that are to be removed from each of the one or more first layers;
- performing a regularization on the one or more filters;
- grouping a set of filters of the one or more filters based on alignment of the set of filters across two or more layers of the neural network;
- aligning structure sparsity across the two or more layers based, at least in part, on the regularization on the one or more filters;
- determining, as a result of the regularization, whether the set of filters satisfy a pruning criteria;
- and
- deactivating one or more second layers of the neural network in response to determining the set of filters satisfies the pruning criteria by pruning the neural network to remove the set of filters from the neural network.
5 more elements
Dependencies are inferred from the claim text.