Systems and methods for performing an autonomous aircraft visual inspection task

A system, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• obtain a template of three-dimensional (3D) representation of a geometry of an aircraft in a first coordinate frame for a visual inspection task to be conducted by an unmanned aerial vehicle (UAV);
• divide the template of the three-dimensional (3D) representation of the geometry of the aircraft into a plurality of landmark regions to obtain a plurality of templates corresponding to each of the plurality of landmark regions, wherein the plurality of templates are obtained in the first coordinate frame;
• register the template of the three-dimensional (3D) representation of the geometry of the aircraft in the first coordinate frame with a target point cloud in a second coordinate frame using a template matching approach, wherein the template matching approach comprises:
• computing a first transformation between a plurality of template point clouds and the target point cloud;
• computing, a matching score between each transformed template point cloud from a plurality of transformed template point clouds and the target point cloud, wherein the matching score is a measure of an average Euclidean distance of a plurality of three-dimensional (3D) points between the plurality of transformed template point clouds and the target point cloud;
• determining an optimal transformed template point cloud from the transformed plurality of template point clouds having an optimal match with the target point cloud based on the computed matching score, wherein the optimal match is determined when the computed matching score between the transformed template point cloud and the t target point cloud is less than a predefined threshold;
• and
• determining a second transformation from the first coordinate frame to the second coordinate frame using the optimal transformed template point cloud, wherein a plurality of first target poses of the UAV are determined for each of the plurality of landmark regions in the second coordinate frame using the determined second transformation;
• sequentially navigate the UAV from a current location to a desired target location of each of the plurality of landmark regions using the plurality of the first target poses of the UAV determined for each of the plurality of landmark regions;
• identify one or more objects of interest in each of the plurality of landmark regions using a deep learning-based object detection approach during sequential navigation, wherein the one or more objects of interest are identified based on one or more characteristics;
• obtain coordinates of bounding boxes of the identified objects of interest on the RGB image plane and corresponding 3D bounding rectangles for each of the bounding boxes using 2D-3D relations;
• calculate a centroid, by extracting RGB-D points that fall within the 3D bounding rectangles;
• compute a set of second target poses in the second coordinate frame for the identified one or more objects of interest in each of the plurality of landmark regions, wherein the set of second target poses are calculated for each centroid of the 3D bounding rectangles, wherein the set of second target poses are at the same height as the centroid and looking towards corresponding centroids from a user-defined distance and wherein the set of second target poses are view poses to capture data for the corresponding objects of interest for inspection;
• and
• autonomously perform the visual inspection task after attaining the computed set of second target poses on the identified one or more objects of interest in each of the plurality of landmark regions, wherein the visual inspection task is performed as a function of area measurement and maximum distance measurement of the identified one or more objects of interest, wherein the RGB-D points captured after reaching to the view based on the second target pose is used by the visual inspection task to get the possible view of the object of interest from the identified one or more objects of interest.

One or more non-transitory computer readable mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• obtaining a template of three-dimensional (3D) representation of a geometry of an aircraft in a first coordinate frame for a visual inspection task to be conducted by an unmanned aerial vehicle (UAV);
• dividing the template of the three-dimensional (3D) representation of the geometry of the aircraft into a plurality of landmark regions to obtain a plurality of templates corresponding to each of the plurality of landmark regions, wherein the plurality of templates are obtained in the first coordinate frame;
• registering the template of the three-dimensional (3D) representation of the geometry of the aircraft in the first coordinate frame with a target point cloud captured by the UAV in a second coordinate frame using a template matching approach, wherein the template matching approach comprises:
• computing a first transformation between a plurality of template point clouds and the target point cloud;
• computing, a matching score between each transformed template point cloud from a plurality of transformed template point clouds and the target point cloud, wherein the matching score is a measure of an average Euclidean distance of a plurality of three-dimensional (3D) points between the plurality of transformed template point clouds and the target point cloud;
• determining an optimal transformed template point cloud from the plurality of transformed template point clouds having an optimal match with the target point cloud based on the computed matching score, wherein the optimal match is determined when the computed matching score between the transformed template point cloud and the target point cloud is less than a predefined threshold;
• and
• determining a second transformation from the first coordinate frame to the second coordinate frame using the optimal transformed template point cloud, wherein a plurality of first target poses of the UAV are determined for each of the plurality of landmark regions in the second coordinate frame using the determined second transformation;
• sequentially navigating the UAV from a current location to a desired target location of each of the plurality of landmark regions using the plurality of the first target poses of the UAV determined for each of the plurality of landmark regions;
• identifying one or more objects of interest in each of the plurality of landmark regions using a deep learning-based object detection approach during sequential navigation, wherein the one or more objects of interest are identified based on one or more characteristics;
• obtaining, coordinates of bounding boxes of the identified objects of interest on the RGB image plane and corresponding 3D bounding rectangles for each of the bounding boxes using 2D-3D relations;
• calculating, a centroid, by extracting RGB-D points that fall within the 3D bounding rectangles;
• computing a set of second target poses in the second coordinate frame for the identified one or more objects of interest in each of the plurality of landmark regions, wherein the set of second target poses are calculated for each centroid of the 3D bounding rectangles, wherein the set of second target poses are at the same height as the centroid and looking towards corresponding centroids from a user-defined distance and wherein the set of second target poses are view poses to capture data for the corresponding objects of interest for inspection;
• and
• autonomously performing the visual inspection task after attaining the computed set of second target poses on the identified one or more objects of interest in each of the plurality of landmark regions, wherein the visual inspection task is performed as a function of area measurement and maximum distance measurement of the identified one or more objects of interest, wherein the RGB-D points captured after reaching to the view based on the second target pose is used by the visual inspection task to get the possible view of the object of interest from the identified one or more objects of interest.