On-device neural networks for natural language understanding

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:

• a projection sequence network configured to receive an input and to generate an output from the input, the projection sequence network comprising:
• a sequence of one or more projection layers, wherein each projection layer has a plurality of projection layer parameters that are learned during training of the projection sequence network, wherein each projection layer is configured to:
• receive a layer input;
• apply a plurality of projection layer functions to the layer input, each projection layer function generating a respective projection function output that projects the layer input to a different space, wherein the plurality of projection layer functions comprise a plurality of locality sensitive hashing functions that were held static during training of the projection neural network, and
• generate a layer output by applying the projection layer parameters for the projection layer to the projection function outputs;
• and
• one or more projection sequence layers positioned structurally after the sequence of one or more projection layers, wherein each of the one or more projection sequence layers is configured to provide first internal state data to a subsequent iteration of such projection sequence layer in a subsequent iteration of the projection sequence network and to receive second internal state data from the subsequent iteration of such projection sequence layer in the subsequent iteration of the projection sequence network;
• wherein execution of the instructions causes the one or more computers to perform operations comprising:
• obtaining the input;
• inputting the input into the projection sequence network;
• and
• receiving the output generated by the projection sequence network.