# US 12,675,909 B2 · Aerial refueling systems and methods > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,675,909 B2 - Document ID: US12675909B2 - Status: Granted - Kind: B2 - Application number: 18/522,002 - Filed: 2023-11-28 - Published: 2025-05-29 - Granted: 2026-07-07 - Earliest claimed: 2023-11-28 - Assignees: The Boeing Company - Applicants: The Boeing Company - Inventors: Fan Hin Hung; Deepak Khosla; Neale Ratzlaff; Tameez Latib; Haden Smith; Leon Nguyen - Web page: https://patentgrep.com/patent/US12675909B2 ## CPC - G06T 7/73: using feature-based methods - G06T 7/75: {involving models} - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/11: Region-based segmentation - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - B64D 39/00: Refuelling during flight - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20132: Image cropping - G06T 2207/30212: Military - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT ## Abstract Disclosed herein are methods, systems, and aircraft for verifying performing automated refueling data. A method includes receiving a two-dimensional (2D) image from a camera, cropping the 2D image based on predefined feature areas of interest of a target object to produce a plurality of cropped images, resizing one or more of the cropped images responsive to the target object being greater than a threshold distance from the camera to produce one or more resized images, determining 2D keypoints of the target object within the one or more resized images or the plurality of cropped images, estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the target object to produce an estimated 6DOF pose, and outputting the 6DOF pose. ## Claims ### Claim 1 A method comprising: receiving a two-dimensional (2D) image from a camera; cropping the 2D image based on predefined feature areas of interest of a target object to produce a plurality of cropped images; resizing one or more of the plurality of cropped images responsive to the one or more of the plurality of cropped images having a resolution different from a training resolution associated with training of a previously trained keypoint detection neural network to produce one or more resized images, the one or more resized images having a resized resolution equal to the training resolution; determining 2D keypoints of the target object within a plurality of input images by inputting the plurality of input images into the keypoint detection neural network, the plurality of input images comprising the one or more resized images; estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the target object to produce an estimated 6DOF pose; and outputting the 6DOF pose; wherein the target object is a receiving aircraft in a close air operation, and wherein the plurality of cropped images comprises a first cropped image and a second cropped image and neither the first cropped image nor the second cropped image is entirely contained within the other. ### Claim 2 (depends on 1) The method of claim 1, wherein each of the cropped images includes a plurality of keypoints. ### Claim 3 (depends on 1) The method of claim 1, wherein the close air operation is a refueling operation. ### Claim 4 (depends on 1) The method of claim 1, further comprising, before producing the plurality of cropped images, performing an initial single crop of the 2D image. ### Claim 5 (depends on 1) The method of claim 1, further comprising performing automated refueling operations responsive to the 6DOF pose. ### Claim 6 (depends on 1) The method of claim 1, wherein: the first cropped image corresponds to a first feature area of interest of the predefined feature areas of interest; the second cropped image corresponds to a second feature area of interest of the predefined feature areas of interest distinct from the first feature area of interest; and the first cropped image does not overlap with the second cropped image. ### Claim 7 A tanker aircraft comprising: a refueling boom; a camera configured to generate a two-dimensional (2D) image of an in-flight refueling operation between a receiver aircraft and the tanker aircraft; a processor; and non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising: cropping the 2D image based on predefined feature areas of interest of the receiver aircraft to produce a plurality of cropped images; resizing one or more of the plurality of cropped images responsive to the one or more of the plurality of cropped images having a resolution different from a training resolution associated with training of a previously trained keypoint detection neural network to produce one or more resized images, the one or more resized images having a resized resolution equal to the training resolution; determining 2D keypoints of the receiver aircraft within a plurality of input images by inputting the plurality of input images into the keypoint detection neural network, the plurality of input images comprising the one or more resized images; estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the receiver aircraft to produce an estimated 6DOF pose; and outputting the 6DOF pose; wherein the plurality of cropped images comprises a first cropped image and a second cropped image and neither the first cropped image nor the second cropped image is entirely contained within the other. ### Claim 8 (depends on 7) The tanker aircraft of claim 7, wherein each of the cropped images includes a plurality of keypoints. ### Claim 9 (depends on 7) The tanker aircraft of claim 7, wherein the code is further configured to cause the processor to perform an initial single crop of the 2D image before producing the plurality of cropped images. ### Claim 10 (depends on 7) The tanker aircraft of claim 7, wherein the tanker aircraft further comprises: an automated refueling system; and a boom operator system; or a pilot director light system, wherein the code is further configured to cause the processor to control operations of the automated refueling system, the boom operator system, or the pilot director light system based on the 6DOF pose of the receiver aircraft. ### Claim 11 (depends on 7) The tanker aircraft of claim 7, wherein: the first cropped image corresponds to a first feature area of interest of the predefined feature areas of interest; the second cropped image corresponds to a second feature area of interest of the predefined feature areas of interest distinct from the first feature area of interest; and the first cropped image does not overlap with the second cropped image. ### Claim 12 A system comprising: a camera configured to generate a two-dimensional (2D) image of an in-flight refueling operation between a receiver aircraft and a tanker aircraft; a processor; and non-transitory computer readable storage media storing code, the code being executable by the processor to perform operations comprising: cropping the 2D image based on predefined feature areas of interest of the receiver aircraft to produce a plurality of cropped images; resizing one or more of the plurality of cropped images responsive to the one or more of the plurality of cropped images having a resolution different from a training resolution associated with training of a previously trained keypoint detection neural network to produce one or more resized images, the one or more resized images having a resized resolution equal to the training resolution; determining 2D keypoints of the receiver aircraft within a plurality of input images by inputting the plurality of input images into the keypoint detection neural network, the plurality of input images comprising the one or more resized images; estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the receiver aircraft to produce an estimated 6DOF pose; and outputting the 6DOF pose; wherein the plurality of cropped images comprises a first cropped image and a second cropped image and neither the first cropped image nor the second cropped image is entirely contained within the other. ### Claim 13 (depends on 12) The system of claim 12, wherein each of the cropped images includes a plurality of keypoints. ### Claim 14 (depends on 12) The system of claim 12, wherein the code is further configured to cause the processor to perform automated refueling operations responsive to the 6DOF pose. ### Claim 15 (depends on 12) The system of claim 12, wherein the code is further configured to generate an uncertainty value of the 6DOF pose. ### Claim 16 (depends on 12) The system of claim 12, wherein: the first cropped image corresponds to a first feature area of interest of the predefined feature areas of interest; the second cropped image corresponds to a second feature area of interest of the predefined feature areas of interest distinct from the first feature area of interest; and the first cropped image does not overlap with the second cropped image.