Solving the distal reward problem through linkage of stdp and dopamine signaling

In a simulated nervous system network having a first pre-neuron and a second post-neuron, and a synaptic pathway between the first pre-neuron and the second post-neuron having synaptic strength (s) and an eligibility trace (c), a method for implementing reinforcement learning in a spiking network based on spike timing dependent plasticity (STDP), comprising:

• (a) firing the first pre-neuron and the second post-neuron within a substantially coincident time of one another to induce changes to the synaptic strength (s) according to a spike-timing-dependent-plasticity (STDP) rule;
• (b) detecting an eligibility trace (c) over a time window commencing with the substantially coincident firings of the first pre-neuron and the second post-neuron, the eligibility trace (c) decaying towards zero over the time window;
• and
• (c) providing an extracellular global diffusive reinforcement signal to the synaptic pathway during the time window and providing an increase in the reinforcement signal at a time in the window occurring at about 1-3 seconds after the coincident firing and commencement of the time window.

In a simulated nervous system network having four random groups of neurons, representing, respectively, an unconditional stimulus (US), a first conditional stimulus (CS1), a second conditional stimulus (CS2), and cortical projections (VTAp) that project to a ventral tegmental area (VTA) of a brain responsible for releasing dopamine, and in which there are synaptic connections from the unconditional stimulus (US) groups of neurons to the cortical projections (VTAp) group of neurons, and from the first conditional stimulus (CS1) group of neurons, and the second conditional stimulus (CS2) group of neurons, a method of shifting the release of dopamine in response to the unconditional stimulus (US) to an earlier reward-predicting conditional stimulus (CS1) and (CS2), comprising:

• (a) initially setting the synaptic connections from the unconditional stimulus (US) groups of neurons to maximum values;
• (b) firing the neurons of the unconditional stimulus (US) groups of neurons to induce changes to the synaptic strength (S) according to a spike-timing-dependent plasticity (STDP) rule, and to cause a response in the neurons of the cortical projections (VTAp);
• (c) firing the neurons of the first conditional stimulus (CS1) prior to firing the unconditional stimulus (US) by about 1±0.25 seconds to induce changes to the synaptic strength (S) in accordance with the STDP rule and to shift to and evoke a response by the cortical projections (VTAp) to the first conditional stimulus (CS1) of neurons;
• and
• (d) firing the neurons of the second conditional stimulus (CS2) prior to firing the neurons of the first conditional stimulus (CS1) by about 1±0.25 seconds to induce changes to the synaptic strength (S) in accordance with the STDP rule and to shift to and evoke a response by the cortical projections (VTAp) to the second conditional stimulus (CS2) of neurons.