Aiding maneuvering of obscured objects

A method of aiding maneuvering of a refueling boom of a refueling aircraft, the method comprising:

• obtaining, by a camera on the refueling aircraft, an image of a receiver aircraft having a receptacle for refueling the receiver aircraft;
• and by a processor on the refueling aircraft;
• determining, from points on the image of the receiver aircraft, an a six-dimensional orientation and position of the receiver aircraft relative to the camera using a pose estimation algorithm and a known relationship between the points on the image of the receiver aircraft and corresponding points on a 3D model of the receiver aircraft;
• determining, based on the orientation and position of the receiver aircraft, a separation distance and respective positions of the refueling boom of the refueling aircraft and the receptacle of the receiver aircraft;
• and
• generating a display on a display device in the refueling aircraft, using the separation distance and respective positions of the refueling boom and the receptacle, the display being of a side perspective view of a 3D model of the refueling boom and the 3D model of the receiver aircraft having the receptacle, illustrating a current position of the refueling boom relative to a current position of the receptacle, from a viewpoint of a virtual camera positioned alongside the refueling boom,
• wherein the display of the side perspective view illustrating the current position of the refueling boom relative to the current position of the receptacle enables an operator on the refueling aircraft to observe and guide the refueling boom to the receptacle;
• wherein generating the display includes retrieving and orienting positions of the 3D model of the refueling boom and the 3D model of the receiver aircraft based on the separation distance and the respective positions of the refueling boom and the receptacle.

A computer-readable storage medium for aiding maneuvering of a refueling boom of a refueling aircraft, the computer-readable storage medium being non-transitory and having computer-readable program code stored therein that in response to execution by a processor, causes a computer system to at least:

• obtain an image of a receiver aircraft having a receptacle for refueling the receiver aircraft from a camera on the refueling aircraft;
• determine, from points on the image of the receiver aircraft, an orientation and position of the receiver aircraft relative to the camera using a pose estimation algorithm and a known relationship between the points on the image of the receiver aircraft and corresponding points on a 3D model of the receiver aircraft;
• determine, based on the orientation and position of the receiver aircraft, a separation distance and respective positions of the refueling boom of the refueling aircraft and the receptacle of the receiver aircraft;
• and
• generate a display on a display device in the refueling aircraft, using the separation distance and respective positions of the refueling boom and the receptacle, the display being of a side perspective view of a 3D model of the refueling boom and the 3D model of the receiver aircraft having the receptacle, illustrating a current position of the refueling boom relative to a current position of the receptacle, from a viewpoint of a virtual camera positioned alongside the refueling boom,
• wherein the display of the side perspective view illustrating the current position of the refueling boom relative to the current position of the receptacle enables an operator on the refueling aircraft to observe and guide the refueling boom to the receptacle;
• wherein the computer system being caused to generate the display includes being caused to retrieve and orient positions of the 3D model of the refueling boom and the 3D model of the receiver aircraft based on the separation distance and the respective positions of the refueling boom and the receptacle.