US 12,690,795 B2 · Reinforcement learning based adaptive state observation for brain- machine interface

Abstract

A reinforcement learning (RL) based adaptive state observation model usable for implementing a brain machine interface (BMI) is proposed for decoding a brain signal to determine a movement action and controlling a machine to perform the movement action. In the model, the brain signal is processed by a neural network (NN) for applying a nonlinear mapping defined by NN weights to the brain signal to thereby yield a transformed brain signal. The NN learns the nonlinear mapping by RL, allowing the weights to be adaptively and continuously updated to follow nonlinearity and non-stationarity of the brain signal. The transformed brain signal is processed by a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action, thereby utilizing the KF to provide smooth generation of the control signal while blocking adverse influence of nonlinearity and non-stationarity of the brain signal to the KF.

Claims20

Dependent claims

A computer-implemented method for controlling a machine to perform a movement action determined by a brain signal of a subject, the method comprising:

processing the brain signal with a neural network (NN) for applying a nonlinear mapping defined by a plurality of weights of the NN to the brain signal to thereby yield a transformed brain signal;
updating the plurality of weights by a reinforcement learning (RL) process such that the NN learns the nonlinear mapping by RL;
and
processing the transformed brain signal with a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action, wherein the control signal is computed as xt|t given by
xt|t=xt|t-1+K(yt−xt|t-1)
where:
xt|t is a posterior estimation of a mean of a state x at time step t, wherein the state x is a state of the KF, the KF being used to model the machine;
xt|t-1 is a prior estimation of the mean of the state x at time step t;
and
K is a Kalman gain given by K=Pt|t-1 (Pt|t-1+R)−1 where is a prior estimation of the covariance of the state x at time step t, and R is a covariance matrix of a Gaussian noise term in yt, in which yt is the transformed brain signal generated by the NN at time step t.

7 more elements

A computer-implemented method for controlling a machine to perform a movement action determined by a brain signal of a subject, the method comprising:

processing the brain signal with a neural network (NN) for applying a nonlinear mapping defined by a plurality of weights of the NN to the brain signal to thereby yield a transformed brain signal;
updating the plurality of weights by a reinforcement learning (RL) process such that the NN learns the nonlinear mapping by RL;
and
processing the transformed brain signal with a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action;
wherein the RL process comprises:
equating the transformed brain signal to be a plurality of scores respectively associated with a plurality of candidate movement actions competing to be the movement action, an individual score of a respective candidate movement action being indicative to a probability that the respective candidate movement action is the movement action;
computing the plurality of weights according to at least the plurality of scores;
and
updating the NN with the computed plurality of weights for configuring the nonlinear mapping;
wherein the computing of the plurality of weights according to at least the plurality of scores comprises:
computing, from the plurality of scores, a plurality of probabilities associated with the plurality of candidate movement actions, wherein an individual probability associated with the respective candidate movement action is the probability that the respective candidate movement action is the movement action;
selecting, from the plurality of candidate movement actions, a winner in competing to be the movement action according to the plurality of probabilities;
determining a reward due to selecting the winner as the movement action according to whether or not the winner is the movement action;
and
computing the plurality of weights according to at least the reward;
wherein P(at=k′), the probability that a k′th candidate movement action in the plurality of candidate movement actions is the movement action, is computed by
P
⁡
(
a
t
=
k
′
)
=
exp
⁡
(
α
⁢
y
k
′
)
∑
k
=
1
N
⁢
exp
⁡
(
α
⁢
y
k
)
where N is a total number of candidate movement actions in the plurality of candidate movement actions, yt=[y1, . . . , yN]T is the transformed brain signal with yk being a kth component of yt, and α≥0 is a controlling parameter;
and
wherein the winner is selected to be a k*th candidate movement action from P(at=k*) among all N probability values of P(αt=k′), k′=1, . . . , N.

49 more elements

Dependencies are inferred from the claim text.

Reinforcement learning based adaptive state observation for brain- machine interface

Life of this patent

Reinforcement learning based adaptive state observation for brain- machine interface

Life of this patent

Abstract

Claims20