Real-time error monitor of receiver aircraft based on machine learning keypoint solutions for automated aerial refueling

A non-transitory, computer-readable medium storing instructions executable by one or more processors of an aircraft controller of an aerial refueling aircraft, the aerial refueling aircraft including a refueling boom and a camera system with one or more digital cameras, the instructions, when executed by the one or more processors, causing the aircraft controller to perform operations comprising:

• receiving, from the camera system, a time-series of images of a receiver aircraft;
• determining, using the time-series of images, an aircraft type of the receiver aircraft and predefined keypoint locations corresponding to the aircraft type;
• detecting, on each aircraft image in the time-series of images using a position estimation machine learning (ML) algorithm, keypoints at the predefined keypoint locations on the receiver aircraft in the aircraft image;
• determining keypoint rates of change for the keypoints in each of the aircraft images by comparing each of the keypoints to a corresponding keypoint in a prior image immediately preceding the aircraft image in the time-series of images;
• estimating, using the keypoint rates of change, predicted keypoint locations for the keypoints in each of the aircraft images;
• determining, via the aircraft controller, a system confidence value by comparing the keypoints in each of the aircraft images with the predicted keypoint locations corresponding to the aircraft image;
• determining if the system confidence value is greater than a threshold confidence value;
• commanding, via the aircraft controller in response to a determination that the system confidence value is greater than a threshold confidence value, movement of the refueling boom in accordance with a predefined boom control protocol;
• and
• commanding, via the aircraft controller in response to a determination that the system confidence value is not greater than the threshold confidence value, movement of the refueling boom to a stowed state.

An aircraft comprising:

• an airframe;
• a rotor assembly and/or a pair of wings attached to the airframe;
• a camera system with one or more digital cameras attached to the airframe;
• a refueling system including a fuel tank and a refueling boom attached to the airframe;
• and
• an aircraft controller communicatively connected to the camera system and the refueling system, the aircraft controller being programmed to:
• receive a time-series of images of a receiver aircraft from the camera system;
• determine, using the time-series of images, an aircraft type of the receiver aircraft and predefined keypoint locations corresponding to the aircraft type;
• detect, on each aircraft image in the time-series of images using a position estimation machine learning (ML) algorithm, keypoints at the predefined keypoint locations on the receiver aircraft in the aircraft image;
• determine keypoint rates of change for the keypoints in each of the aircraft images by comparing each of the keypoints to a corresponding keypoint in a prior image immediately preceding the aircraft image in the time-series of images;
• estimate, using the keypoint rates of change, predicted keypoint locations for the keypoints in each of the aircraft images;
• determine a system confidence value by comparing the keypoints in each of the aircraft images with the predicted keypoint locations corresponding to the aircraft image;
• in response to a determination that the system confidence value is greater than a threshold confidence value, control movement of the refueling boom in accordance with a predefined boom control protocol;
• and
• in response to a determination that the system confidence value is not greater than the threshold confidence value, control movement of the refueling boom to a stowed state.