Apparatus and methods for reinforcement learning in artificial neural networks

A controller apparatus comprising a non-transitory storage medium, said non-transitory storage medium configured to implement reinforcement learning in a neural network comprising a plurality of units, said non-transitory storage medium comprising a plurality of instructions configured to, when executed:

• evaluate a network performance measure at a first time instance;
• identify at least one unit of said plurality of units, said identified at least one unit characterized by an activity characteristic meeting a criterion;
• and
• potentiate, based at least in part on said network performance measure being below a threshold of said identified at least one unit said potentiation characterized by an increase in said activity characteristic;
• wherein said activity characteristic comprises a probability of said identified at least one unit generating a spiking output based on a spiking input;
• and
• said potentiation is determined based at least in part on said identified at least one unit being previously inactive in response to receiving said spiking input.

A method of adjusting an efficacy of a synaptic connection configured to provide an input into a spiking neuron of a computerized spiking network, said method comprising:

• based at least in part on (i) a negative reward indication, and (ii) provision of said input to said spiking neuron at a time, increasing said efficacy;
• where said provision of said input is characterized by an absence of neural output within a time window relative to said time;
• said negative reward indication is configured based at least in part on a performance measure determined based at least in part on a present performance and a target performance associated with a process;
• said negative reward indication determined based at least in part on said present performance being outside a predetermined measure from said target performance;
• said increasing said efficacy is based at least in part on said negative reward indication and is effectuated based at least in part on said input received via an interface;
• and
• wherein said increasing said efficacy is determined based at least in part on said absence of neural output within a time window relative to said time.

A robotic apparatus having a plant and a controller, said robotic apparatus configured to:

• the controller comprising a processor configured to;
• identify an undesirable result of an action of said plant of said robotic apparatus;
• and
• for said controller, perform at least one of:
• (i) penalize at least one input source of a plurality of possible input sources that contributed to said undesirable result;
• and/or
• (ii) potentiate at least a portion of said plurality of possible input sources that did not contribute to said undesirable result;
• where:
• said penalization is configured based at least in part on a performance measure determined based at least in part on a present performance and a target performance associated with a process;
• said penalization comprises a negative reward determined based at least in part on said present performance being outside a predetermined measure from said target performance;
• where said potentiation is based at least in part on a reinforcement signal and is effectuated based at least in part on a spiking input received via said at least one input source;
• said potentiation is configured to increase a probability of a spiking neuron generating a spiking output based on said spiking input;
• and
• said potentiation is determined based at least in part on said spiking neuron being previously inactive in response to receiving said spiking input.

A computerized spiking neuron apparatus, said computerized spiking neuron apparatus comprising:

• means for modifying a learning parameter based at least in part on a reinforcement signal and a quantity relating to a first adjustment and second adjustment:
• wherein:
• said reinforcement signal is configured based at least in part on a performance measure determined based at least in part on a present performance and a target performance associated with a process;
• said reinforcement signal comprises a negative reward determined based at least in part on said present performance being outside a predetermined measure from said target performance;
• said means for modifying said learning parameter is based at least in part on said reinforcement signal and is effectuated based at least in part on a spiking input received via an interface;
• said second adjustment is configured to increase a probability of said computerized spiking neuron generating a spiking output based on said spiking input:
• and
• said second adjustment is determined based at least in part on said computerized spiking neuron being previously inactive in response to receiving said spiking input.