Generative neural networks with compressive attention layers

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a network input;
• and
• processing the network input to generate an output sequence of tokens that comprises a respective output token at each of a plurality of output positions, the processing comprising, for each of the output positions:
• processing the output token at the preceding output position using a generative neural network conditioned on the network input to generate the output token at the output position, wherein the generative neural network comprises a compressive attention layer, and wherein the processing comprises, for the compressive attention layer:
• identifying a memory state for the compressive attention layer as of the output position;
• obtaining a layer input token for the compressive attention layer that is generated from the output token at the preceding output position;
• generating, from the layer input token for the compressive attention layer, a query vector, a key vector, and a value vector;
• performing a set of one or more optimization steps on a compression objective using the key vector and the value vector to update the memory state for the compressive attention layer;
• and
• after updating the memory state for the compressive attention layer, applying the memory state to the query vector to generate a layer output token for the compressive attention layer.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a network input;
• and
• processing the network input to generate an output sequence of tokens that comprises a respective output token at each of a plurality of output positions, the processing comprising, for each of the output positions:
• processing the output token at the preceding output position using a generative neural network conditioned on the network input to generate the output token at the output position, wherein the generative neural network comprises a compressive attention layer, and wherein the processing comprises, for the compressive attention layer:
• identifying a memory state for the compressive attention layer as of the output position;
• obtaining a layer input token for the compressive attention layer that is generated from the output token at the preceding output position;
• generating, from the layer input token for the compressive attention layer, a query vector, a key vector, and a value vector;
• performing a set of one or more optimization steps on a compression objective using the key vector and the value vector to update the memory state for the compressive attention layer;
• and
• after updating the memory state for the compressive attention layer, applying the memory state to the query vector to generate a layer output token for the compressive attention layer.