System and method for vision-based flight self-stabilization by deep gated recurrent Q-networks

A non-transitory computer-readable medium having computer program for, when being executed by a processor, performing a method for vision-based self-stabilization by deep gated recurrent Q-networks (DGRQNs) for unmanned aerial vehicles (UAVs), the method comprising:

• receiving a plurality of raw images captured by a camera installed on the UAV;
• receiving an initial reference image for stabilization and obtaining an initial camera pose from the initial reference image;
• extracting a fundamental matrix between consecutive images and estimating a current camera pose relative to the initial camera pose, wherein the camera pose includes an orientation and a location of the camera;
• based on the estimated current camera pose, predicting an action to counteract a lateral disturbance of the UAV based on the DGRQNs, comprising:
• encoding the estimated camera pose into a feature vector;
• processing the feature vector by deep neuron networks (DNNs);
• processing the feature vector by deep gated recurrent neural networks (DGRNs);
• and
• processing the feature vector by Q-learning to obtain a difference between the estimated current camera pose and the initial camera pose;
• and
• based on the predicted action to counteract the lateral disturbance of the UAV, driving the UAV back to the initial camera pose.

A system for vision-based self-stabilization by deep gated recurrent Q-networks (DGRQNs) for unmanned aerial vehicles (UAVs), comprising:

• a memory;
• and
• a processor coupled to the memory, wherein the processor is configured to:
• receive a plurality of raw images captured by a camera installed on the UAV, extract a fundamental matrix between consecutive images, and estimate a current camera pose relative to an initial camera pose, wherein the camera pose includes an orientation and a location of the camera;
• and
• based on the estimated current camera pose, predict an action to counteract a lateral disturbance of the UAV based on DGRQNs, comprising:
• encode the estimated camera pose into a feature vector;
• process the feature vector by deep neuron networks (DNNs);
• process the feature vector by deep gated recurrent neural networks (DGRNs);
• and
• process the feature vector by Q-learning to obtain a difference between the estimated current camera pose and the initial camera pose;
• and
• based on the predicted action to counteract the lateral disturbance of the UAV, drive the UAV back to the initial camera pose.