Method for acquiring a pseudo-3d box from a 2d bounding box by regression analysis and learning device and testing device using the same

A method for acquiring a pseudo-3D box bounding an object in a test image, comprising steps of:

• (a) a testing device acquiring or supporting another device to acquire the test image including an object for testing bounded by a 2D bounding box for testing, on condition that (i) a learning device acquires or supports another device to acquire a training image including an object for training bounded by a 2D bounding box for training;
• (ii) the learning device performs or supports another device to perform (ii-1) a process of classifying a pseudo-3D orientation of the object for training, by referring to information on probabilities corresponding to respective patterns of pseudo-3D orientation, and (ii-2) a process of acquiring 2D coordinates of vertices of the pseudo-3D box for training by using regression analysis;
• and (iii) the learning device adjusts or supports another device to adjust parameters thereof by backpropagating loss information determined by referring to at least one of (iii-1) differences between the acquired 2D coordinates of the vertices of the pseudo-3D box for training and 2D coordinates of ground truth corresponding to the pseudo-3D box for training, and (iii-2) differences between the classified pseudo-3D orientation for training and ground truth corresponding to the pseudo-3D orientation for training;
• (b) the testing device performing or supporting another device to perform (i) a process of classifying a pseudo-3D orientation of the object for testing, by referring to information on probabilities corresponding to respective patterns of pseudo-3D orientation and (ii) a process of acquiring 2D coordinates of vertices of the pseudo-3D box for testing by using the regression analysis,
• wherein the 2D coordinates define 2D coordinates of vertices of the pseudo-3D box,
• the object being a vehicle, and the pseudo-3D box being configured to contain the object in an autonomous driving application, and
• at the process of (ii) in the step of (b), the testing device performs or supports another device to perform a process of calculating (b-1) (i) a displacement between a center of the 2D bounding box for testing and a first point to be used as a center of a first target 2D box which corresponds to a first surface of the pseudo-3D box for testing, and (ii) a displacement between the center of the 2D bounding box for testing and a second point to be used as a center of a second target 2D box which corresponds to a second surface, which is antipodal to the first surface, of the pseudo-3D box for testing;
• (b-2) respective ratios of (i) a width of the first target 2D box to a width of the 2D bounding box for testing, (ii) a width of the second target 2D box to the width of the 2D bounding box for testing, (iii) a height of the first target 2D box to a height of the 2D bounding box for testing, and (iv) a height of the second target 2D box to the height of the 2D bounding box for testing;
• and (b-3) coordinates of P5, P6, P7, and P8 which consist vertices of the first target 2D box, and coordinates of P1, P2, P3, and P4 which consist vertices of the second target 2D box, by referring to the displacements and the ratios, wherein the P8 is a vertex of the 2D bounding box for testing and the P1 is a vertex of the 2D bounding box for testing diagonal to the P8, wherein the P4 and the P5 are not in contact with line segments of the 2D bounding box for testing.

A learning device for acquiring a pseudo-3D box from a 2D bounding box in a training image, comprising:

• a processor configured to
• obtain the training image including an object bounded by the 2D bounding box;
• and
• perform or support another device to perform (i) a process of classifying a pseudo-3D orientation of the object, by referring to information on probabilities corresponding to respective patterns of pseudo-3D orientation;
• (ii) a process of acquiring 2D coordinates of vertices of the pseudo-3D box by using regression analysis;
• and (iii) a process of adjusting parameters thereof by backpropagating loss information determined by referring to at least one of (iii-1) differences between the acquired 2D coordinates of the vertices of the pseudo-3D box and 2D coordinates of ground truth corresponding to the pseudo-3D box, and (iii-2) differences between the classified pseudo-3D orientation and ground truth corresponding to the pseudo-3D orientation,
• wherein the 2D coordinates define 2D coordinates of vertices of the pseudo-3D box,
• the object being a vehicle, and the pseudo-3D box being configured to contain the object in an autonomous driving application, and
• the processor performs or supports another device to perform a process of calculating (i) a first vector from a center of the 2D bounding box to determine an end point of the first vector as a center of a first target 2D box which corresponds to a first surface of the pseudo-3D box, and (ii) a second vector from the center of the 2D bounding box to determine an end point of the second vector as a center of a second target 2D box which corresponds to a second surface, which is antipodal to the first surface, of the pseudo-3D box by regression;
• respective ratios of (i) a width of the first target 2D box to a width of the 2D bounding box, (ii) a width of the second target 2D box to the width of the 2D bounding box, (iii) a height of the first target 2D box to a height of the 2D bounding box, and (iv) a height of the second target 2D box to the height of the 2D bounding box by the regression;
• and coordinates of P5, P6, P7, and P8 which consist vertices of the first target 2D box, and coordinates of P1, P2, P3, and P4 which consist vertices of the second target 2D box, by referring to the displacements and the ratios, wherein the P8 is a vertex of the 2D bounding box and the P1 is a vertex of the 2D bounding box diagonal to the P8, and wherein the P4 and the P5 are not in contact with line segments of the 2D bounding box.

A testing device for acquiring a pseudo-3D box bounding an object in a test image, comprising:

• a processor configured to
• obtain the test image including an object for testing bounded by a 2D bounding box for testing, on condition that (i) a learning device acquires or supports another device to acquire a training image including an object for training bounded by a 2D bounding box for training;
• (ii) the learning device performs or supports another device to perform (ii-1) a process of classifying a pseudo-3D orientation of the object for training, by referring to information on probabilities corresponding to respective patterns of pseudo-3D orientation and (ii-2) a process of acquiring 2D coordinates of vertices of the pseudo-3D box for training by using regression analysis;
• and (iii) the learning device adjusts or supports another device to adjust parameters thereof by backpropagating loss information determined by referring to at least one of (iii-1) differences between the acquired 2D coordinates of the vertices of the pseudo-3D box for training and 2D coordinates of ground truth corresponding to the pseudo-3D box for training, and (iii-2) differences between the classified pseudo-3D orientation for training and ground truth corresponding to the pseudo-3D orientation for training;
• and
• perform or support another device to perform (i) a process of classifying a pseudo-3D orientation of the object for testing, by referring to information on probabilities corresponding to respective patterns of pseudo-3D orientation and (ii) a process of acquiring 2D coordinates of vertices of the pseudo-3D box for testing, by using the regression analysis,
• wherein the 2D coordinates define 2D coordinates of vertices of the pseudo-3D box,
• the object being a vehicle, and the pseudo-3D box being configured to contain the object in an autonomous driving application, and
• the processor performs or supports another device to perform a process of calculating (i) a first vector from a center of the 2D bounding box for testing to determine an end point of the first vector as a center of a first target 2D box for testing which corresponds to a first surface of the pseudo-3D box for testing, and (ii) a second vector from the center of the 2D bounding box for testing to determine an end point of the second vector as a center of a second target 2D box for testing which corresponds to a second surface, which is antipodal to the first surface, of the pseudo-3D box for testing;
• and respective ratios of (i) a width of the first target 2D box for testing to a width of the 2D bounding box for testing, (ii) a width of the second target 2D box for testing to the width of the 2D bounding box for testing, (iii) a height of the first target 2D box for testing to a height of the 2D bounding box for testing, and (iv) a height of the second target 2D box for testing to the height of the 2D bounding box for testing, and coordinates of P5, P6, P7, and P8 which are vertices of the first target 2D box for testing, and coordinates of P1, P2, P3, and P4 which are vertices of the second target 2D box for testing, by referring to the displacements and the ratios, wherein the P8 is a vertex of the 2D bounding box for testing and the P1 is a vertex of the 2D bounding box for testing diagonal to the P8, and wherein the P4 and the P5 are not in contact with line segments of the 2D bounding box for testing.