Augmenting neural networks with external memory

An augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising:

• a neural network, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the neural network output comprises:
• a system output portion,
• a write portion,
• a write vector, and
• a respective read key for each of one or more read heads;
• an external memory;
• and
• a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the neural network outputs:
• providing an output derived from the system output portion as a system output in the sequence of system outputs;
• determining a respective writing weight for each of a plurality of locations in the external memory from the write portion of the neural network output;
• writing data defined by the write vector to the external memory in accordance with the respective writing weights;
• maintaining a temporal link matrix that tracks a history of writing weights for previous writing operations performed on the external memory by the memory interface subsystem, wherein each element in the temporal link matrix represents a degree to which a corresponding first location was written to after a corresponding second location by a previous writing operation for a previous neural network output;
• updating the temporal link matrix based on the writing weights for the neural network output;
• for each of one or more read heads:
• determining a respective content-based reading weight for each of the plurality of locations in the external memory from the read key for the read head,
• determining one or more respective history weights for each of the plurality of locations in the external memory from the updated temporal link matrix,
• determining a respective final reading weight for each of the plurality of locations from the one or more history weights and the content-based reading weight, and
• reading data from the external memory in accordance with the final reading weights;
• and
• combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

A method of processing a sequence comprising a plurality of system inputs to generate a sequence comprising a plurality of system outputs, the method comprising, for each of the system inputs:

• obtaining data previously read from an external memory during processing of a preceding system input;
• combining the data previously read from the external memory with the system input to generate a neural network input;
• providing the neural network input as input to a neural network, wherein the neural network is configured to process the neural network input to generate a neural network output from the neural network input, wherein the neural network output comprises:
• a system output portion,
• a read portion,
• a write key, and
• a write vector;
• determining a respective content-based weight for each of a plurality of locations in the external memory from the write key;
• determining a respective allocation weight for each of the plurality of locations in the external memory based on how available each of the plurality of locations in the external memory is for a modification of the data stored in the location, comprising:
• determining a respective usage value for each of the locations that represents a degree to which the availability of the location is being used, comprising, for each of the locations, increasing a previous usage value for the location based on a previous final writing weight used to write to the location for a previous neural network output, and
• determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations;
• determining the respective allocation weights for the locations based at least in part on the respective usage values for the locations;
• determining a respective final writing weight for each of the plurality of locations in the external memory from the respective content-based weight for the location and the respective allocation weight for the location;
• writing data defined by the write vector to the external memory in accordance with the final writing weights;
• determining one or more sets of reading weights for each of the plurality of locations in the external memory from the read portion;
• reading data from the external memory in accordance with the sets of reading weights;
• and
• providing an output derived from the system output portion as a system output in the sequence of system outputs.