Robust lidar-to-camera sensor alignment

A method for performing a lidar to camera alignment algorithm comprising:

• detecting, by a lidar, a depth point cloud of a first field of view including a first object and a second object;
• generating a first control point in response to a location of the first object within the depth point cloud and a second control point in response to a location of the second object within the depth point cloud;
• capturing, by a camera, an image of a second field of view including a third object;
• generating a third control point in response to a location of the third object detected in response to the image;
• calculating a first reprojection error in response to the first control point and the third control point and a second reprojection error in response to the second control point and the third control point;
• determining an alignment pair in response to a lowest reprojection error between the first reprojection error and the second reprojection error wherein the alignment pair with the lowest reprojection error are determined to be corresponding objects;
• generating a lidar to camera extrinsic parameter between the first object location and the third object location in response to the first reprojection error in response to the first reprojection error being less than the second reprojection error;
• generating an output alignment in response to the lidar to camera extrinsic parameter;
• and
• controlling a vehicle in response to the output alignment and a subsequent image captured by the camera.

A vehicle control system comprising:

• a lidar sensor for generating a point cloud representative a three dimensional space;
• a camera for generating an image;
• a processor for detecting a first object location and a second object location in response to the point cloud, detecting a third object location in response to the image, calculating a first reprojection error between the first object location and the third object location and a second reprojection error between the second object location and the third object location, determining an alignment pair in response to a lowest reprojection error between the first reprojection error and the second reprojection error wherein the alignment pair with the lowest reprojection error are determined to be corresponding objects, generating a translation and a rotation between the alignment pair in response to the first reprojection error and in response to the first object and the third object being determined to be the alignment pair, and generating an output alignment in response to the translation and the rotation;
• and
• a vehicle controller for controlling the host vehicle in response to the output alignment and a subsequent image captured by the camera.