Systems and methods for calculating refueling tanker boom 3D position for aerial refueling

A tanker aircraft comprising:

• a refueling boom;
• a camera configured to generate a two-dimensional (2D) image of the refueling boom;
• a processor;
• and
• non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
• receiving the two-dimensional (2D) image from the camera;
• determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom;
• determining a 6 degree-of-freedom (6DOF) pose using the determined 2D keypoints and corresponding three-dimensional (3D) space keypoints;
• optimizing 3D keypoints associated with moveable components of the refueling boom by iteratively solving for a plurality of boom control parameters for the moveable components, based on a difference between the determined 2D keypoints and projected 2D keypoints generated by projecting the 3D keypoints onto the 2D image using the determined 6DOF pose to produce optimized 3D keypoints such that the projected 2D keypoints substantially align with the determined 2D keypoints;
• and
• estimating a position of a tip of the refueling boom based on the 6DOF pose and on the optimized 3D keypoints.

A refueling system comprising:

• a processor;
• and
• non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising:
• receiving a two-dimensional (2D) image from a camera;
• determining 2D keypoints of a refueling boom located within the 2D image based on a predefined point model of the refueling boom;
• determining a 6 degree-of-freedom (6DOF) pose using the determined 2D keypoints and corresponding three-dimensional (3D) keypoints;
• optimizing 3D keypoints associated with moveable components of the refueling boom by iteratively solving for a plurality of boom control parameters for the moveable components, based on a difference between the determined 2D keypoints and projected 2D keypoints generated by projecting the 3D keypoints onto the 2D image using the determined 6DOF pose to produce optimized 3D keypoints such that the projected 2D keypoints substantially align with the determined 2D keypoints;
• and
• estimating a position of a tip of the refueling boom based on the 6DOF pose and on the optimized 3D keypoints.