Complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system, method, and vehicular device

A complex network cognition-based FRL end-to-end autonomous driving control method, comprising the following steps:

• step 1:
• building an urban dense traffic simulation environment in a CARLA simulator, wherein the simulation environment contains a driving subject, a traffic participants, and a road infrastructure,
• the driving subject is a plurality of agents, modeled as Markov decision processes, respectively, and using a reinforcement learning module for a steering wheel, a throttle and a brake control, the Markov decision process is described by a tuple (S, A, P, R, γ), wherein S denotes a state set, corresponding to state quantities acquired by a measurement encoder and an image encoder, and contains a steering wheel angle, a throttle, a brake, a gear, lateral and longitudinal speeds, and a 15-channel semantic BEV;
• A denotes an action set, corresponding to the steering wheel, the throttle, and the brake control quantities of the driving subject;
• P denotes a state transfer equation p:
• S×A→P(S), each state-action pair (s, α)∈S×A has a probability distribution p(·|s, α) of entering a new state after adopting an action α, in a state s;
• R denotes a reward function R:
• S×S×A→R, R (St+1, st, αt) denotes a reward obtained after entering the new state st+1 from an original state st, a goodness of performing the action is defined by the reward function;
• γ denotes a discount factor, γ∈[0, 1], is configured to compute a cumulative reward η(πθ)=Σi=0Tγiri, wherein T denotes a current moment, γi denotes a discount factor of moment i, and ri denotes an immediate reward of moment i, a solution to the Markov decision process is to find a strategy π:
• S→A maximize the cumulative reward π*:
• =argmaxθ π(πθ), the reinforcement learning module integrates implicit state quantities output by the measurement encoder and the image encoder and outputs a corresponding optimal control policy;
• step 2:
• building the complex network cognition module to model a driving situation of the driving subject, establish a complex network model, and output dynamic driving suggestions based on state quantities provided by the measurement encoder through an activation function, the complex network model represents a dynamic relationship between nodes within a field range through a variable Gaussian safety field based on risk center transfer, the nodes contain the driving subject, the traffic participants, and the road infrastructure;
• step 3:
• constructing an end-to-end neural network, comprising 2 fully connected layers used by the measurement encoder, 6 convolutional layers used by the image encoder and 6 fully connected layers used by the reinforcement learning module, the neural network has two output heads, an action head and a value head, the action head outputs two parameters of a beta distribution and the value head outputs a value of the action;
• step 4:
• interacting the driving subject with a CARLA simulation environment and storing experiences in respective local replay buffers, wherein when the number of samples reaches a certain threshold, the samples are sampled from the respective local replay buffer according to a mini-batch, and then neural network parameters are updated according to a designed loss function;
• step 5:
• uploading the neural network parameters corresponding to each driving subject to a federated learning module, aggregating global parameters based on the multiple neural network parameters according to an aggregation interval, and sending the global parameters to each agent until the network converges;
• wherein in step 1, the traffic participants comprise other vehicles and pedestrians, the road infrastructure contains traffic lights and traffic signs, represented in the image encoder input using a 4-frame semantic BEV;
• in step 2, the complex network cognition module takes different activation functions according to different driving suggestions:
• Activate
• go
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 300
• /
• Risk
• )
• )
• -
• 1
• Activate
• stop
• (
• Risk
• )
• =
• 4
• (
• 1
• +
• exp
• (
• -
• 0.2
• *
• Risk
• )
• )
• -
• 1
• in step 2, the dynamic driving suggestions are represented using the beta distribution:
• B
• risk
• =
• B
• (
• Activate
• go
• (
• Risk
• )
• ,
• β
• ,
• go
• B
• risk
• =
• B
• ⁡
• (
• α
• ,
• Activate
• stop
• (
• Risk
• )
• )
• ,
• stop
• in step 3, the neural network of the measurement encoder uses a Relu activation function for the 2 fully connected layers, the neural network of the image encoder uses the Relu activation function for the 5 convolutional layers except for the last convolutional layer that spreads the state volume without using the activation function, the neural network of the reinforcement learning module uses a Softplus activation function for the last layer of the action head, the last layer of the value head does not use the activation function, the other fully connected layers use the Relu activation function;
• in step 4, a set of parameters used in the training process:
• a learning rate 0.00001;
• a total step size 12288;
• a mini-batch sampling size 256;
• a set of loss function weights λppo 0.5, λexp 0.01, and λrisk 0.05, respectively;
• a range of a PPO clip 0.2;
• and a pair of parameters, γ 0.99 and λ 0.9, for a generalized advantage estimation;
• in step 4, the loss function used for the training process uses the dynamic risk suggestions based loss risk, a guidance of the agent is realized by calculating a KL-divergence of the strategy and the dynamic driving suggestions output by the driving subject Nz=100 steps before the termination state;
• in step 5, the federated learning module is a multi-agent framework that uses a local replay buffer architecture between agents;
• and
• in step 5, a global parameter aggregation uses a parameter-averaged aggregation method with the aggregation interval of 256.