Augmenting neural networks with external memory using reinforcement learning

A method of processing a sequence of system inputs to generate a sequence of system outputs performed by one or more computers, the method comprising:

• providing a neural network input to a neural network, wherein the neural network is configured to receive the neural network input and to process the neural network input to generate a neural network output from the neural network input, the neural network output comprising:
• a system output portion that is used to generate a system output in the sequence of system outputs that corresponds to the neural network input,
• a reinforcement learning portion that is used to make a hard selection of one of a plurality of memory access processes for accessing an external memory that is external to the neural network, wherein making the hard selection comprises making a hard selection of a partition to be accessed from a plurality of partitions of the external memory, wherein the selected partition comprises a plurality of memory locations in the external memory, and
• a differentiable portion that defines a content of a write to the plurality of locations in the selected partition of the external memory, wherein the reinforcement learning portion, the differentiable portion, and the system output portion are different portions of the neural network output;
• providing an output derived from the system output portion of the neural network output as the system output in the sequence of system outputs;
• selecting the memory access process from the predetermined set of memory access processes for accessing the external memory from the reinforcement learning portion of the neural network output, wherein the selection of the memory access process is a non-differentiable operation comprising:
• the hard selection of the one of the partitions of the external memory based on the reinforcement learning portion of the neural network output, and wherein the neural network has been trained to generate neural network outputs that are used to select memory access processes through reinforcement learning;
• writing and reading data only from the locations in the selected partition of the external memory using the differentiable portion of the neural network output;
• and
• combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for processing a sequence of system inputs to generate a sequence of system outputs, the operations comprising:

• providing a neural network input to a neural network, wherein the neural network is configured to receive the neural network input and to process the neural network input to generate a neural network output from the neural network input, the neural network output comprising:
• a system output portion that is used to generate a system output in the sequence of system outputs that corresponds to the neural network input,
• a reinforcement learning portion that is used to make a hard selection of one of a plurality of memory access processes for accessing an external memory that is external to the neural network, wherein making the hard selection comprises making a hard selection of a partition to be accessed from a plurality of partitions of the external memory, wherein the selected partition comprises a plurality of memory locations in the external memory, and
• a differentiable portion that defines a content of a write to the plurality of locations in the selected partition of the external memory, wherein the reinforcement learning portion, from the differentiable portion, and the system output portion are different portions of the neural network output;
• providing an output derived from the system output portion of the neural network output as the system output in the sequence of system outputs;
• selecting the memory access process from the predetermined set of memory access processes for accessing the external memory from the reinforcement learning portion of the neural network output, wherein the selection of the memory access process is a non-differentiable operation comprising:
• the hard selection of the one of the partitions of the external memory based on the reinforcement learning portion of the neural network output, and wherein the neural network has been trained to generate neural network outputs that are used to select memory access processes through reinforcement learning;
• writing and reading data only from the locations in the selected partition of the external memory using the differentiable portion of the neural network output;
• and
• combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.