Multi-stage computationally efficient neural network inference

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a first network input;
• processing the first network input using a smaller neural network to generate a first network output for the first network input that comprises a respective first score for each of a first plurality of categories;
• determining, from the first network output for the first network input, whether the first network output for the first network input is likely to be inaccurate;
• in response to determining that the first network output for the first network input is likely to be inaccurate:
• providing the first network input as input to a larger neural network that is configured to process the first network input to generate a second network output for the first network input that comprises a respective second score for each of a second plurality of categories, the second plurality of categories comprising all of the categories in the first plurality of categories, wherein the larger neural network is more computationally expensive than the smaller neural network;
• and
• classifying the first network input based on the second network output for the first network input;
• obtaining a second network input;
• processing the second network input using the smaller neural network to generate a first network output for the second network input that comprises a respective first score for each of the first plurality of categories;
• determining, from the first network output for the second network input, whether the first network output for the second network input is likely to be inaccurate;
• and
• in response to determining that the first network output for the second network input is not likely to be inaccurate:
• classifying second first network input based on the first network output for the second network input without providing the first network input as input to the larger neural network.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a first network input;
• processing the first network input using a smaller neural network to generate a first network output for the first network input that comprises a respective first score for each of a first plurality of categories;
• determining, from the first network output for the first network input, whether the first network output for the first network input is likely to be inaccurate;
• in response to determining that the first network output for the first network input is likely to be inaccurate:
• providing the first network input as input to a larger neural network that is configured to process the first network input to generate a second network output for the first network input that comprises a respective second score for each of a second plurality of categories, the second plurality of categories comprising all of the categories in the first plurality of categories, wherein the larger neural network is more computationally expensive than the smaller neural network;
• and
• classifying the first network input based on the second network output for the first network input;
• obtaining a second network input;
• processing the second network input using the smaller neural network to generate a first network output for the second network input that comprises a respective first score for each of the first plurality of categories;
• determining, from the first network output for the second network input, whether the first network output for the second network input is likely to be inaccurate;
• and
• in response to determining that the first network output for the second network input is not likely to be inaccurate:
• classifying second first network input based on the first network output for the second network input without providing the first network input as input to the larger neural network.