Complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system, method, and vehicular device

A complex network cognition-based FRL end-to-end autonomous driving control method, comprising the following steps:

• step 1:
• building an urban dense traffic simulation environment in a CARLA simulator, wherein the simulation environment contains a driving subject, traffic participants, and a road infrastructure,
• the driving subject is a plurality of agents, modeled as Markov decision processes, respectively, and using a reinforcement learning module implemented by at least one processor for a steering wheel, a throttle and a brake control, the Markov decision process is described by a tuple (S, A, P, R, γ), wherein S denotes a state set, corresponding to state quantities acquired by a measurement encoder implemented by the at least one processor and an image encoder implemented by the at least one processor, and contains a steering wheel angle, a throttle, a brake, a gear, lateral and longitudinal speeds, and a 15-channel semantic BEV;
• A denotes an action set, corresponding to the steering wheel, the throttle, and the brake control quantities of the driving subject;
• P denotes a state transfer equation p:
• S×A→P(S), each state-action pair (s, a)∈S×A has a probability distribution p(·|s, a) of entering a new state after adopting an action a, in a state s;
• R denotes a reward function R:
• S×X×A→R, R(st+1, st, at) denotes a reward obtained after entering the new state st+1 from an original state st, a goodness of performing the action is defined by the reward function;
• γ denotes a discount factor, γ∈[0, 1], is configured to compute a cumulative reward η(πθ)=Σi=0Tγiri, wherein T denotes a current moment, γi denotes a discount factor of moment i, and ri denotes an immediate reward of moment i, a solution to the Markov decision process is to find a strategy π:
• S→A maximize the cumulative reward π*:=argmaxθη(πθ), the reinforcement learning module integrates implicit state quantities output by the measurement encoder and the image encoder and outputs a corresponding optimal control policy;
• step 2:
• building the complex network cognition module implemented by the at least one processor to model a driving situation of the driving subject, establish a complex network model, and output dynamic driving suggestions based on state quantities provided by the measurement encoder through an activation function, the complex network model represents a dynamic relationship between nodes within a field range through a variable Gaussian safety field based on risk center transfer, the nodes contain the driving subject, the traffic participants, and the road infrastructure;
• step 3:
• constructing an end-to-end neural network, comprising 2 fully connected layers used by the measurement encoder, 6 convolutional layers used by the image encoder and 6 fully connected layers used by the reinforcement learning module, the neural network has two output heads, an action head and a value head, the action head outputs two parameters of a beta distribution and the value head outputs a value of the action;
• step 4:
• interacting the driving subject with a CARLA simulation environment and storing experiences in respective local replay buffers, wherein when the a number of samples reaches a certain threshold, samples of the number of samples are sampled from the respective local replay buffer according to a mini-batch, and then neural network parameters are updated according to a designed loss function;
• step 5:
• uploading the neural network parameters corresponding to each driving subject to a federated learning module implemented by the at least one processor, aggregating global parameters based on the neural network parameters according to an aggregation interval, and sending the global parameters to each agent until the neural network converges;
• wherein in step 1, the traffic participants comprise other vehicles and pedestrians, the road infrastructure contains traffic lights and traffic signs, represented in the image encoder input using a 4-frame semantic BEV;
• in step 2, the complex network cognition module takes different activation functions according to different driving suggestions:
• Activate
• go
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 300
• /
• Risk
• )
• )
• -
• 1
• Activate
• stop
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 0.2
• *
• Risk
• )
• )
• -
• 1
• in step 2, the dynamic driving suggestions are represented using the beta distribution:
• B
• risk
• =
• B
• (
• Activate
• go
• (
• Risk
• )
• ,
• β
• ,
• go
• B
• risk
• =
• B
• ⁡
• (
• α
• ,
• Activate
• stop
• (
• Risk
• )
• )
• ,
• stop
• in step 3, the neural network of the measurement encoder uses a Relu activation function for the 2 fully connected layers, the neural network of the image encoder uses the Relu activation function for the 5 convolutional layers except for the last convolutional layer that spreads a state volume without using any activation function, the neural network of the reinforcement learning module uses a Softplus activation function for a last layer of the action head, a last layer of the value head does not use any activation function, the other fully connected layers use the Relu activation function;
• in step 4, a set of parameters used in the training process:
• a learning rate 0.00001;
• a total step size 12288;
• a mini-batch sampling size 256;
• a set of loss function weights λppo 0.5, λexp 0.01, and λrisk 0.05, respectively;
• a range of a PPO clip 0.2;
• and a pair of parameters, γ 0.99 and λ 0.9, for a generalized advantage estimation;
• in step 4, the loss function used for the training process uses dynamic risk suggestions based loss risk, a guidance of the agent is realized by calculating a KL-divergence of the strategy and the dynamic driving suggestions output by the driving subject Nz=100 steps before a termination state;
• in step 5, the federated learning module is a multi-agent framework that uses a local replay buffer architecture between agents;
• and
• in step 5, a global parameter aggregation uses a parameter-averaged aggregation method with the aggregation interval of 256.

A vehicular device, wherein the vehicular device includes A complex network cognition-based FRL end-to-end autonomous driving control system comprising:

• a measurement encoder, an image encoder, a complex network cognition module, a reinforcement learning module, and a federated learning module, wherein:
• the measurement encoder is implemented by at least one processor and is configured to obtain state quantities required by the complex network cognition module, which is implemented by the at least one processor, and the reinforcement learning module, which is implemented by the at least one processor, the state quantities required by the complex network cognition module comprise a x-coordinate, a y-coordinate, a heading angle change and a speed of a given driving agent of a plurality of driving agents, the state quantities required by the complex network cognition module are handed over to the complex network cognition module as inputs, the state quantities required by the reinforcement learning module comprise a steering wheel angle, a throttle, a brake, a gear, a lateral speed and a longitudinal speed, the state quantities required by the reinforcement learning module are given to the reinforcement learning module after extracting features from a two-layer fully connected network;
• the image encoder is implemented by the at least one processor and is configured to obtain an amount of image implicit state required by the reinforcement learning module, an image used is a 15-channel semantic bird's eye view (BEV), iRL∈[0, 1]192*192*15, 192 is in pixels and the BEV used is 5 px/m, 15 channels contain a drivable domain, a desired path, a road edge, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs, wherein the desired path is calculated using an A* algorithm, the semantic BEV is extracted by multilayer convolutional layers to extract implicit features and then passed to the reinforcement learning module as another part of the inputs;
• the complex network cognition module is configured to model a driving situation of a driving subject, and to obtain a maximum risk value of the driving subject in a current driving situation according to the state quantity provided by the measurement encoder, and finally to output dynamic driving suggestions based on a risk value through an activation function;
• the reinforcement learning module is configured to integrate the state quantities output from the measurement encoder and the image encoder, output corresponding strategies according to integrated network inputs, and interact with an environment to generate experience samples stored in a local replay buffer in the federated learning module, which is implemented by the at least one processor, when a number of the experience samples reaches a certain threshold, a batch of sample is taken from the local replay buffer for training, and finally trained neural network parameters are uploaded to the federated learning module;
• and
• the federated learning module is configured to receive the finally trained neural network parameters uploaded by the reinforcement learning module of the plurality of driving agents, and to aggregate a set of global parameters based on the finally trained neural network parameters, and finally to send the global parameters to the plurality of driving agents until a neural network converges, a global parameter aggregation is performed by a following equation:
• ϕ
• m
• *
• =
• 1
• N
• ⁢
• ∑
• n
• ϕ
• m
• n
• wherein ϕm* denotes the global parameters at time m, N denotes a number of the plurality of driving agents, and ϕmn denotes the neural network parameters at time m of an nth driving agent of the plurality of driving agents;
• wherein the activation function is configured to map the risk value, Activate(Risk) represents different activation functions according to different driving suggestions, and the mapped risk value will be used as a basis for guiding an output strategy of the reinforcement learning module:
• Activate
• go
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 300
• /
• (
• Risk
• )
• )
• -
• 1
• Activate
• stop
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 0.2
• *
• (
• Risk
• )
• )
• -
• 1
• wherein Activatego(Risk) denotes an activation function when a driving suggestion is forward, Activatestop(Risk) denotes an activation function when the driving suggestion is stop, and Risk denotes a current risk value of a self-vehicle, a dynamic risk suggestion Brisk:
• B
• r
• ⁢
• i
• ⁢
• s
• ⁢
• k
• =
• B
• ⁢
• (
• Activate
• g
• ⁢
• o
• (
• Risk
• )
• ,
• β
• )
• ,
• go
• B
• r
• ⁢
• i
• ⁢
• s
• ⁢
• k
• =
• B
• ⁢
• (
• α
• ,
• Activate
• stop
• (
• Risk
• )
• )
• ,
• stop
• wherein B denotes a beta distribution with αstop=βgo=1.