Real-time control selection and calibration using neural network

A method for real-time control selection and calibration in a vehicle using a deep-Q network (DQN), the method comprising:

• measuring, by one or more sensors disposed on the vehicle, real-time static and dynamic data about the vehicle;
• altering static and dynamic characteristics of the vehicle with one or more actuators disposed on the vehicle;
• utilizing one or more control modules each having a processor, a memory, and input/output (I/O) ports in communication with the one or more sensors and the one or more actuators, the processor executing program code portions stored in the memory, the program code portions:
• causing the one or more sensors and the one or more actuators to obtain vehicle dynamics information and road surface estimation information;
• generating a vehicle dynamical context from vehicle dynamics information and road surface estimation information;
• deciding which one of a plurality of predefined calibrations is appropriate for the vehicle dynamical context, wherein deciding which one of a plurality of predefined calibrations is appropriate for the vehicle dynamical context further comprises:
• refining the vehicle dynamical context by passing the vehicle dynamical context through the DON, the DON having multiple fully-connected layers:
• generating an expected Q-value Q(a,θ), where a is the calibration selected, and θ is a weight to be learned;
• and
• tuning the θ such that a function:
• ∥Q′(a,θ)−(r+γmaxaQ(a,θ)∥ is minimized, where γ is a discounting factor that increases with time, and r(n) is a reward received;
• wherein r(n) is large when the calibration selected achieves a desired vehicle dynamics state, and r(n) is small when the calibration selected does not achieve the desired vehicle dynamics state;
• generating an actuator command to the one or more actuators based on a selected one of the plurality of predefined calibrations;
• and
• continuously and recursively obtaining the vehicle dynamics and road surface estimation information, generating the vehicle dynamical context, deciding which one of the plurality of predefined calibrations is appropriate, and generating an actuator command based on a selected one of the plurality of predefined calibrations while the vehicle is being operated.

A system for real-time control selection and calibration in a vehicle using a deep-Q network (DQN), the system comprising:

• one or more sensors disposed on the vehicle, the one or more of sensors measuring real-time static and dynamic data about the vehicle;
• one or more actuators disposed on the vehicle, the one or more actuators altering static and dynamic characteristics of the vehicle;
• one or more control modules each having a processor, a memory, and input/output (I/O) ports in communication with the one or more sensors and the one or more actuators, the processor executing program code portions stored in the memory, the program code portions comprising:
• a first program code portion that causes the one or more sensors and the one or more actuators to obtain vehicle dynamics information and road surface estimation information;
• a second program code portion that utilizes the vehicle dynamics information and road surface estimation information to generate a vehicle dynamical context, wherein the vehicle dynamical context is generated by passing the vehicle dynamics information and the road surface information through a recurrent convolutional neural network (RCNN), the RCNN including a plurality of model predictive control (MPC) modules, each of the plurality of MPC modules defining program code portions having a predictor and a controller, the predictor containing actuator settings, and the controller executing the actuator settings of the predictor to produce an actuator command;
• and a plurality of actuator calibrations stored in memory and executed by a single controller to produce an actuator command;
• a third program code portion that decides which one of a plurality of predefined calibrations is appropriate for the vehicle dynamical context;
• wherein the third program code portion decides which one of the plurality of predefined calibrations is appropriate for a current vehicle dynamical context by passing the vehicle dynamical context through the DQN, applying a key performance indicator (KPI) reward r(n) to the vehicle dynamical context and selecting a calibration having a maximum possible value among the plurality of predefined calibrations;
• partitions vehicle dynamics information and road surface information into episodic time frames having a predefined duration;
• evaluates calibration selection decisions based on performance indices;
• and computes a weighted sum of performance indices as the reward r(n), wherein the performance indices comprise:
• tracking error, energy consumption, and current vehicle dynamic state information;
• wherein the third program code portion refines the vehicle dynamical context by passing the vehicle dynamical context through the DQN, the DQN having multiple fully-connected layers;
• generates an expected Q-value Q(a,θ), where a is the calibration selected, and θ is a weight to be learned;
• and tunes the θ such that a function:
• ∥Q′(a,θ)−(r+γmaxaQ(a,θ))∥ is minimized, where γ is a discounting factor that increases with time, and r(n) is a reward received;
• wherein r(n) is large when the calibration selected achieves a desired vehicle dynamics state, and r(n) is small when the calibration selected does not achieve the desired vehicle dynamics state;
• a fourth program code portion that generates an actuator command to the one or more actuators based on the one of a selected one of the plurality of predefined calibrations;
• and
• a fifth program code portion that continuously and recursively causes the one or more sensors and the one or more actuators to send vehicle dynamics and road surface estimation information to second program code portion and causes the third, fourth, and fifth program code portions to execute while the vehicle is being operated.