# US 2026/0242067 A1 · Real-time error monitor of receiver aircraft based on machine learning keypoint solutions for automated aerial refueling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0242067 A1
- Document ID: US20260242067A1
- Status: Published application
- Kind: A1
- Application number: 19/644,278
- Filed: 2026-04-10
- Published: 2026-08-20
- Earliest claimed: 2024-04-23
- Assignees: The Boeing Company
- Applicants: The Boeing Company
- Inventors: Balram KANDORIA
- Web page: https://patentgrep.com/patent/US20260242067A1

## CPC

- B64D 39/00: Refuelling during flight
- G06T 7/251: {involving models}
- G06T 7/75: {involving models}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of operating an aerial refueling aircraft includes an aircraft controller receiving a time-series of images of a receiver aircraft from an aircraft camera system and, using these images, determines an aircraft type and predefined keypoint locations for the receiver aircraft. Using a position estimation machine learning algorithm, the controller modifies each aircraft image to place keypoints at the aircraft keypoints on the receiver aircraft. The controller determines a rate of change for each keypoint in each image and, using these keypoint rates of change, predicts keypoint locations for the keypoints in each image. The controller determines a system confidence value by comparing the keypoints in each image with its corresponding predicted keypoint location. Responsive to determining that the system confidence value exceeds a threshold confidence value, the controller controls movement of the aircraft's refueling boom in accordance with a predefined boom control protocol.

## Claims

### Claim 1

A method of operating an aerial refueling aircraft having a refueling boom and a camera system, the method comprising:
receiving, via an aircraft controller from the camera system, a time-series of images of a receiver aircraft;
detecting, on each aircraft image in the time-series of images via the aircraft controller using a position estimation machine learning (ML) algorithm, respective keypoints on the receiver aircraft;
determining, via the aircraft controller, keypoint rates of change for the respective keypoints in each aircraft image in the time-series of images;
estimating, via the aircraft controller using the keypoint rates of change, predicted keypoint locations for the respective keypoints in each aircraft image in the time-series of images;
determining, via the aircraft controller, a system confidence value by comparing the respective keypoints in each aircraft image in the time-series of images with the predicted keypoint locations corresponding to the aircraft image;
commanding, via the aircraft controller in response to a determination that the system confidence value is greater than a threshold confidence value, movement of the refueling boom in accordance with a predefined boom control protocol; and
commanding, via the aircraft controller in response to a determination that the system confidence value is not greater than the threshold confidence value, movement of the refueling boom to a stowed state.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting the respective keypoints in each aircraft image in the time-series of images includes generating a set of keypoints in a two-dimensional (2D) pixel space.

### Claim 3 (depends on 2)

The method of claim 2, further comprising converting, via the aircraft controller using a Perspective-and-Point (PnP) iterative model, the set of keypoints in the 2D pixel space to a set of three-dimensional (3D) keypoints in a 3D coordinate space and mapped onto a 3D model of the receiver aircraft.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
determining a respective set of six-degrees-of-freedom (6DoF) values for each aircraft image in the time-series of images using a respective prior image preceding each aircraft image in the time-series of images, the respective set of 6DoF values including estimated velocity and acceleration values; and
determining an aircraft-calibrated flight envelope corresponding to an aircraft type of the receiver aircraft, the aircraft-calibrated flight envelope including predefined maximum velocity and acceleration values.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
comparing the estimated velocity and acceleration values to the predefined maximum velocity and acceleration values to determine if the respective set of 6DoF values are within the aircraft-calibrated flight envelope; and
commanding, via the aircraft controller in response to a determination that the respective set of 6DoF values are not within the aircraft-calibrated flight envelope, movement of the refueling boom to the stowed state.

### Claim 6 (depends on 3)

The method of claim 3, further comprising labelling, via the aircraft controller using the PnP iterative model, each keypoint of the respective keypoints in each aircraft image in the time-series of images as either an inlier or an outlier, wherein each keypoint is labelled as the outlier when, based on a best fit analysis, the keypoint has a worse fit to the 3D model of the receiver aircraft than other keypoints of the respective keypoints.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the keypoint rates of change for the respective keypoints in each aircraft image in the time-series of images includes:
determining a temporal offset between the aircraft images in the time-series of images; and
determining, using the temporal offset, a respective velocity and acceleration for each keypoint of the respective keypoints.

### Claim 8 (depends on 1)

The method of claim 1, wherein estimating the predicted keypoint locations for the respective keypoints in each aircraft image in the time-series of images includes:
identifying, for each keypoint of the respective keypoints, a corresponding keypoint in a respective prior image preceding each aircraft image in the time-series of images; and
calculating, using a kinematic displacement equation, a respective predicted keypoint location for each keypoint of the respective keypoints as a function of the keypoint and a corresponding keypoint rate of change.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining the system confidence value includes calculating an object keypoint similarity (OKS) as:
OKS
=
e
-
d
i
2
2
⁢
s
2
⁢
k
i
2
wherein:
di is a Euclidean distance between a temporally predicted keypoint and a KP-predicted keypoint;
s is a scalar for an area of a bounding box in the aircraft image divided by a total image area of the aircraft image; and
k is a per-keypoint constant that controls fall off.

### Claim 10 (depends on 1)

The method of claim 1, wherein determining the system confidence value includes calculating a pixel space error and a cartesian space error.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
determining, via the aircraft controller using the time-series of images, an aircraft type of the receiver aircraft and predefined keypoint locations corresponding to the aircraft type,
wherein the predefined keypoint locations are unique to the aircraft type,
wherein the predefined keypoint locations include an aircraft engine keypoint, an aircraft nose cone keypoint, and an aircraft fuel port, and
wherein the respective keypoints in each aircraft image in the time-series of images are detected at the predefined keypoint locations.

### Claim 12

An aircraft comprising:
an airframe;
a rotor assembly or a pair of wings attached to the airframe;
a camera system with one or more digital cameras attached to the airframe;
a refueling system including a fuel tank and a refueling boom attached to the airframe; and
an aircraft controller communicatively connected to the camera system and the refueling system, the aircraft controller being programmed to:
receive a time-series of images of a receiver aircraft from the camera system;
detect, on each aircraft image in the time-series of images using a position estimation machine learning (ML) algorithm, respective keypoints on the receiver aircraft;
determine keypoint rates of change for the respective keypoints in each aircraft image in the time-series of images;
estimate, using the keypoint rates of change, predicted keypoint locations for the respective keypoints in each aircraft image in the time-series of images;
determine a system confidence value by comparing the respective keypoints in each aircraft image in the time-series of images with the predicted keypoint locations corresponding to the aircraft image;
in response to a determination that the system confidence value is greater than a threshold confidence value, control movement of the refueling boom in accordance with a predefined boom control protocol; and
in response to a determination that the system confidence value is not greater than the threshold confidence value, control movement of the refueling boom to a stowed state.

### Claim 13 (depends on 12)

The aircraft of claim 12, wherein, to detect the respective keypoints in each aircraft image in the time-series of images, the aircraft controller is programmed to:
generate a set of keypoints in a two-dimensional (2D) pixel space; and
convert, using a Perspective-and-Point (PnP) iterative model, the set of keypoints in the 2D pixel space to a set of three-dimensional (3D) keypoints in a 3D coordinate space and mapped onto a 3D model of the receiver aircraft.

### Claim 14 (depends on 13)

The aircraft of claim 13, wherein the aircraft controller is further programmed to:
label, using the PnP iterative model, each keypoint of the respective keypoints in each aircraft image in the time-series of images as either an inlier or an outlier, wherein each keypoint is labelled as the outlier when, based on a best fit analysis, the keypoint has a worse fit to the 3D model of the receiver aircraft than other keypoints of the respective keypoints.

### Claim 15 (depends on 12)

The aircraft of claim 12, wherein the aircraft controller is further programmed to:
determine a respective set of six-degrees-of-freedom (6DoF) values for each aircraft image in the time-series of images using a respective prior image preceding each aircraft image in the time-series of images, the respective set of 6DoF values including estimated velocity and acceleration values; and
determine an aircraft-calibrated flight envelope corresponding to an aircraft type of the receiver aircraft, the aircraft-calibrated flight envelope including predefined maximum velocity and acceleration values.

### Claim 16 (depends on 15)

The aircraft of claim 15, wherein the aircraft controller is further programmed to:
compare the estimated velocity and acceleration values to the predefined maximum velocity and acceleration values to determine if the respective set of 6DoF values are within the aircraft-calibrated flight envelope; and
in response to a determination that the respective set of 6DoF values are not within the aircraft-calibrated flight envelope, command movement of the refueling boom to the stowed state.

### Claim 17 (depends on 12)

The aircraft of claim 12, wherein, to determine the keypoint rates of change for the respective keypoints in each aircraft image in the time-series of images, the aircraft controller is programmed to:
determine a temporal offset between the aircraft images in the time-series of images; and
determine, using the temporal offset, a respective velocity and acceleration for each keypoint of the respective keypoints.

### Claim 18 (depends on 12)

The aircraft of claim 12, wherein, to determine the system confidence value, the aircraft controller is programmed to calculate an object keypoint similarity (OKS) as:
OKS
=
e
-
d
i
2
2
⁢
s
2
⁢
k
i
2
wherein:
di is a Euclidean distance between a temporally predicted keypoint and a KP-predicted keypoint;
s is a scalar for an area of a bounding box in the aircraft image divided by a total image area of the aircraft image; and
k is a per-keypoint constant that controls fall off.

### Claim 19 (depends on 12)

The aircraft of claim 12, wherein, to estimate the predicted keypoint locations for the respective keypoints in each aircraft image in the time-series of images, the aircraft controller is programmed to:
identify, for each keypoint of the respective keypoints, a corresponding keypoint in a respective prior image preceding each aircraft image in the time-series of images; and
calculate, using a kinematic displacement equation, a respective predicted keypoint location for each keypoint of the respective keypoints as a function of the keypoint and a corresponding keypoint rate of change.

### Claim 20

A non-transitory, computer-readable medium storing instructions executable by one or more processors of an aircraft controller of an aerial refueling aircraft, the aerial refueling aircraft including a refueling boom and a camera system with one or more digital cameras, the instructions, when executed by the one or more processors, causing the aircraft controller to perform operations comprising:
receiving, via an aircraft controller from the camera system, a time-series of images of a receiver aircraft;
detecting, on each aircraft image in the time-series of images via the aircraft controller using a position estimation machine learning (ML) algorithm, respective keypoints on the receiver aircraft;
determining, via the aircraft controller, keypoint rates of change for the respective keypoints in each aircraft image in the time-series of images;
estimating, via the aircraft controller using the keypoint rates of change, predicted keypoint locations for the respective keypoints in each aircraft image in the time-series of images;
determining, via the aircraft controller, a system confidence value by comparing the respective keypoints in each aircraft image in the time-series of images with the predicted keypoint locations corresponding to the aircraft image;
commanding, via the aircraft controller in response to a determination that the system confidence value is greater than a threshold confidence value, movement of the refueling boom in accordance with a predefined boom control protocol; and
commanding, via the aircraft controller in response to a determination that the system confidence value is not greater than the threshold confidence value, movement of the refueling boom to a stowed state.
