Augmenting neural networks with external memory

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for processing a sequence of system inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network and an external memory, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the external memory is external to the neural network and is configured to store a respective value vector in each of a plurality of locations in the external memory, and wherein the operations comprise, for each neural network output:

• providing an output derived from a first portion of the neural network output as a system output in the sequence of system outputs;
• determining one or more sets of writing weights for each of the plurality of locations in the external memory from a second portion of the neural network output, wherein the second portion includes a content-based subportion and a location-based subportion, wherein the content-based subportion is different from the location-based subportion, and wherein determining each of the one or more sets of writing weights comprises:
• for each of the plurality of locations in the external memory:
• computing a similarity measure between (i) a key vector derived from the content-based subportion and (ii) a respective value vector stored in the location, and
• determining a respective content-based writing weight based on the computed similarity measure, and
• adjusting the content-based writing weights using preceding writing weights assigned to the plurality of locations and a shift vector derived from the location-based subportion to generate the one or more sets of writing weights;
• writing data defined by a third portion of the neural network output to the external memory in accordance with the sets of writing weights;
• determining one or more sets of reading weights for each of the plurality of locations in the external memory from a fourth portion of the neural network output;
• reading data from the external memory in accordance with the sets of reading weights;
• and
• combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

A system comprising one or more computers and one or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for processing a sequence of system inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network and an external memory, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the external memory is external to the neural network and is configured to store a respective value vector in each of a plurality of locations in the external memory, and wherein the operations comprise, for each neural network output:

• providing an output derived from a first portion of the neural network output as a system output in the sequence of system outputs;
• determining one or more sets of writing weights for each of the plurality of locations in the external memory from a second portion of the neural network output, wherein the second portion includes a content-based subportion and a location-based subportion, wherein the content-based subportion is different from the location-based subportion, and wherein determining each of the one or more sets of writing weights comprises:
• for each of the plurality of locations in the external memory:
• computing a similarity measure between (i) a key vector derived from the content-based subportion and (ii) a respective value vector stored in the location, and
• determining a respective content-based writing weight based on the computed similarity measure, and
• adjusting the content-based writing weights using preceding writing weights assigned to the plurality of locations and a shift vector derived from the location-based subportion to generate the one or more sets of writing weights;
• writing data defined by a third portion of the neural network output to the external memory in accordance with the sets of writing weights;
• determining one or more sets of reading weights for each of the plurality of locations in the external memory from a fourth portion of the neural network output;
• reading data from the external memory in accordance with the sets of reading weights;
• and
• combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.