RGBD-based neural network and training for pose estimation

A tanker aircraft comprising:

• a camera system comprising an imaging camera configured to generate a two-dimensional (2D) image of a receiving aircraft and a depth camera configured to generate a depth image of the receiving aircraft, wherein the depth camera is a different type of camera than the imaging camera, and wherein the depth image is generated independently of the 2D image;
• a processor;
• and
• non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
• detecting the receiving aircraft within the 2D image based on 2D image values;
• determining 2D keypoints of the second device located within the 2D image based on the depth image and the 2D image values by inputting the 2D image values and the depth image into a neural network keypoint model, wherein the neural network keypoint model is trained on a plurality of training depth images and a plurality of training 2D image values;
• determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints;
• and
• outputting a position of at least a component of the receiving aircraft based on the 6DOF pose.

A refueling system comprising:

• a processor;
• and
• non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
• receiving a two-dimensional (2D) image from an imaging camera of an imaging device of a first device;
• detecting a second device within the 2D image based on 2D image values;
• receiving a depth image from a depth camera of the imaging device, wherein the depth camera is a different type of camera than the imaging camera, and wherein the depth image is generated independently of the 2D image;
• determining 2D keypoints of the second device located within the 2D image based on the depth image and the 2D image values by inputting the 2D image values and the depth image into a neural network keypoint model, wherein the neural network keypoint model is trained on a plurality of training depth images and a plurality of training 2D image values;
• determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints;
• and
• outputting a position of at least a component of the second device based on the 6DOF pose.