Increasing sparsity to improve neural network efficiency

A system comprising:

• one or more computers;
• and
• one or more computer-readable media storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• storing, by the one or more computers, parameter values of parameter matrices of one or more layers of a neural network, wherein the parameter values of the parameter matrices comprise (i) weight values of the one or more layers of the neural network, and (ii) predictor values that have been trained to predict levels of importance of items processed by the neural network, wherein the weight values and the predictor values are trained jointly during training of the neural network;
• and
• generating, by the one or more computers, an output of the neural network based on an input token for the neural network, wherein generating the output of the neural network comprises generating an output of the one or more layers, and wherein generating the output of the one or more layers comprises:
• determining, by the one or more computers, a value for each of multiple items using the predictor values, wherein determining the values comprises generating a vector of values by computing a multiplication involving a predictor matrix comprising the predictor values and a query vector that is based on an input to the one or more layers;
• selecting, by the one or more computers, a proper subset of the items based on the values in the vector, wherein selecting the proper subset comprises (i) determining a threshold level based on a calculation using the values and (ii) applying a soft thresholding operator based on the determined threshold to obtain a sparse vector representing the selected subset of the items;
• and
• generating, by the one or more computers, output of the one or more layers including by limiting computation based on the selected proper subset using the sparse vector.

One or more non-transitory computer-readable media storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to perform operations comprising:

• storing, by the one or more computers, parameter values of parameter matrices of one or more layers of a neural network, wherein the parameter values of the parameter matrices comprise (i) weight values of the one or more layers of the neural network, and (ii) predictor values that have been trained to predict levels of importance of items processed by the neural network, wherein the weight values and the predictor values are trained jointly during training of the neural network;
• and
• generating, by the one or more computers, an output of the neural network based on an input token for the neural network, wherein generating the output of the neural network comprises generating an output of the one or more layers, and wherein generating the output of the one or more layers comprises:
• determining, by the one or more computers, a value for each of multiple items using the predictor values, wherein determining the values comprises generating a vector of values by computing a multiplication involving a predictor matrix comprising the predictor values and a query vector that is based on an input to the one or more layers;
• selecting, by the one or more computers, a proper subset of the items based on the values in the vector, wherein selecting the proper subset comprises (i) determining a threshold level based on a calculation using the values and (ii) applying a soft thresholding operator based on the determined threshold to obtain a sparse vector representing the selected subset of the items;
• and
• generating, by the one or more computers, output of the one or more layers including by limiting computation based on the selected proper subset using the sparse vector.