Augmented recurrent neural network with external memory

A method for processing one or more inputs to generate a sequence of system outputs using an augmented neural network system comprising a recurrent neural network,

• wherein the recurrent neural network is configured to receive a network input at each of a plurality of time steps and to process the network input and a current internal state of the recurrent neural network to generate a neural network output for the time step and to update the current internal state of the recurrent neural network, and
• wherein the method comprises, for each of the plurality of time steps:
• providing an output derived from the neural network output for the time step as a system output for the time step;
• determining, from the neural network output for the time step, memory state parameters for the time step;
• updating a current state of an external memory using the memory state parameters for the time step, wherein the external memory is configured to store data vectors written to the external memory, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:
• writing, based on the memory state parameters determined from the neural network output for the time step, data to one or more new rows of the matrix memory;
• and
• updating, based on the memory state parameters determined from the neural network output for the time step, the size vector, comprising:
• adding a respective new entry to the size vector for each new row of the matrix memory;
• reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:
• generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and
• reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights;
• and
• combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement an augmented neural network system for processing one or more inputs to generate a sequence of system outputs, the augmented neural network system comprising:

• a recurrent neural network, wherein the recurrent neural network is configured to receive a network input at each of a plurality of time steps and to process the network input and a current internal state of the recurrent neural network to generate a neural network output for the time step and to update the current internal state of the recurrent neural network;
• and
• a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the plurality of time steps:
• providing an output derived from the neural network output for the time step as a system output for the time step;
• determining, from the neural network output for the time step, memory state parameters for the time step;
• updating a current state of an external memory using the memory state parameters for the time step, wherein the external memory is configured to store data vectors written to the external memory by a memory interface subsystem, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:
• writing, based on the memory state parameters determined from the neural network output for the time step, data to one or more new rows of the matrix memory;
• and
• updating, based on the memory state parameters determined from the neural network output for the time step, the size vector, comprising:
• adding a respective new entry to the size vector for each new row of the matrix memory;
• reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:
• generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and
• reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights;
• and
• combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.