Apparatus and method for performing heterogeneous sensor fusion

A method comprising:

• detecting a first object by processing detection points input from at least one of a lidar or a radar corresponding to a first sensor;
• detecting a second object by processing an image input from a camera corresponding to a second sensor;
• obtaining converted coordinates of the detection points by converting coordinates of the detection points in a first coordinate system of the first sensor into coordinates in a second coordinate system of the second sensor using a trans-matrix obtained via a calibration between the first coordinate system and the second coordinate system;
• calculating a matching rate by matching the detection points of the converted coordinates with the second object of the image;
• determining whether the first object and the second object are identical based on the calculated matching rate;
• generating track information by fusing information from the first sensor and the second sensor when the first object and the second object are identical;
• and
• outputting the track information,
• wherein the calculating the matching rate comprises:
• generating a first box corresponding to the first object projected onto an image layer, generating a second box corresponding to the second object, and calculating a value of the matching rate of an area of an overlapping region between the first box and the second box to an area of the first box or the second box, or
• generating a first box corresponding to the first object projected onto an image layer, generating a second box corresponding to the second object, and calculating a value of the matching rate of a number of detection points included in the second box to a total number of the whole detection points.

A non-transitory computer-readable recording medium having a program recorded thereon, the program to direct a processor to perform acts of:

• detecting a first object by processing detection points input from at least one of a lidar or a radar corresponding to a first sensor;
• detecting a second object by processing an image input from a camera corresponding to a second sensor;
• obtaining converted coordinates of the detection points by converting coordinates of the detection points in a first coordinate system of the first sensor into coordinates in a second coordinate system of the second sensor using a trans-matrix obtained via a calibration between the first coordinate system and the second coordinate system;
• calculating a matching rate by matching the detection points of the converted coordinates with the second object of the image;
• determining whether the first object and the second object are identical based on the calculated matching rate;
• generating track information by fusing information from the first sensor and the second sensor when the first object and the second object are identical;
• and
• outputting the track information,
• wherein the calculating the matching rate comprises:
• generating a first box corresponding to the first object projected onto an image layer, generating a second box corresponding to the second object, and calculating a value of the matching rate of an area of an overlapping region between the first box and the second box to an area of the first box or the second box, or
• generating a first box corresponding to the first object projected onto an image layer, generating a second box corresponding to the second object, and calculating a value of the matching rate of a number of detection points included in the second box to a total number of the whole detection points.

A vehicle comprising:

• a first sensor configured to acquire detection points corresponding to a first object around the vehicle, the detection points being input from at least one of a lidar or a radar corresponding to the first sensor;
• a camera corresponding to a second sensor configured to acquire an image of a region around the vehicle;
• and
• a heterogeneous sensor fusion apparatus configured to fuse information from the first sensor and the second sensor,
• wherein the heterogeneous sensor fusion apparatus is further configured to:
• detect the first object by processing detection points input from the first sensor;
• detect a second object by processing an image input from the second sensor;
• obtaining converted coordinates of the detection points by converting coordinates of the detection points in a first coordinate system of the first sensor into coordinates in a second coordinate system of the second sensor using a trans-matrix obtained via a calibration between the first coordinate system and the second coordinate system;
• calculate a matching rate by matching the detection points of the converted coordinates with the second object of the image;
• determine whether the first object and the second object are identical based on the calculated matching rate;
• generate track information by fusing the information from the first sensor and the second sensor when it is determined that the first object and the second object are identical;
• and
• output the generated track information,
• wherein the heterogeneous sensor fusion apparatus is further configured to:
• generate a first box corresponding to the first object projected onto an image layer, generate a second box corresponding to the second object, and calculate a value of the matching rate of an area of an overlapping region between the first box and the second box to an area of the first box or the second box, or
• generate a first box corresponding to the first object projected onto an image layer, generate a second box corresponding to the second object, and calculate a value of the matching rate of a number of detection points included in the second box to a total number of the whole detection points.