US 11,080,544 B1 · Method and device for calibrating pitch of camera on vehicle and method and device for continual learning of vanishing point estimation model to be used for calibrating the pitch

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
and
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
wherein, at the step of (c), the calibration device performs or supports another device to perform a process of instructing the pitch-deciding module to (i) select a third target object among the objects by referring to the object profiling information, (ii) calculate a second height of the third target object by using the third pitch, (iii) validate the third pitch by confirming whether the second height is within a height range, (iv) (iv-1) if the third pitch is determined as valid, output the third pitch as the decided pitch, and (iv-2) if the third pitch is determined as invalid, compare the first target object corresponding to the first pitch and the second target object corresponding to the second pitch, and (iv-2-a) if the first target object and the second target object are determined as identical, output one of the first pitch and the second pitch as the decided pitch and (iv-2-b) if the first target object and the second target object are determined as not identical, output a specific pitch as the decided pitch, wherein the specific pitch corresponds to a specific target object, whose lateral distance from the specific vehicle is smaller than that of a remaining target object, among the first target object and the second target object, and (v) if none of the objects and the lanes are detected from the driving image, output the fourth pitch as the decided pitch.

2 more elements

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
(d) the calibration device performing or supporting another device to perform a process of calculating one or more pitch losses by referring to the decided pitch and the fourth pitch and a process of on-vehicle continual learning of the VP estimation network by using the pitch losses;
wherein the calibration device performs or supports another device to perform (i) during the driving operation of the specific vehicle, a process of fast-adapting by instance-wise incremental learning of the VP estimation network using the pitch losses and (ii) after completion of the driving operation of the specific vehicle, a process of balanced continual learning of the VP estimation network using sampled data sampled during the driving operation, to thereby recover from catastrophic forgetting due to the process of fast-adapting.

2 more elements

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
and
(d) the calibration device performing or supporting another device to perform a process of calculating one or more pitch losses by referring to the decided pitch and the fourth pitch and a process of on-vehicle continual learning of the VP estimation network by using the pitch losses;
wherein the calibration device performs or supports another device to perform (i) during the driving operation of the specific vehicle, a process of fast-adapting by instance-wise incremental learning of the VP estimation network using the pitch losses, (ii) a process of transmitting the sampled data, used during the instance-wise incremental learning, to a learning server, to thereby allow the learning server to (ii-1) execute server-side continual learning of a reference VP estimation network, corresponding to the VP estimation network, by using the sampled data and (ii-2) transmit one or more parameters, updated through the server-side continual learning, to the calibration device, and (iii) after completion of the driving operation of the specific vehicle, a process of updating the VP estimation network by using the parameters received from the learning server.

3 more elements

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
(d) the calibration device performing or supporting another device to perform a process of calculating one or more pitch losses by referring to the decided pitch and the fourth pitch and a process of on-vehicle continual learning of the VP estimation network by using the pitch losses;
and
(e) the calibration device performing or supporting another device to perform (i) a process of transmitting a trained VP estimation network model, created by the on-vehicle continual learning, to a learning server, to thereby allow the learning server to (i-1) evaluate the trained VP estimation network model and at least one another trained VP estimation network model received from at least one another vehicle and thus (i-2) select a best VP estimation network model and transmit the best VP estimation network model to the calibration device and (ii) a process of updating the VP estimation network by using the best VP estimation network model received from the learning server.

3 more elements

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
and
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
wherein, at the step of (b), the calibration device performs or supports another device to perform a process of instructing the object-based pitch estimation module to (i) select a second specific object, whose lateral distance is smallest, as the first target object, wherein the second specific object is selected among second candidate objects (1) whose lateral distances from the specific vehicle are equal to or less than a second distance threshold, (2) whose bounding boxes are not truncated, (3) whose object classes are a vehicle class, and (4) which are not occluded, by referring to the object profiling information, (ii) acquire an average height of the first target object by referring to detection history of the first target object, and (iii) (iii-1) if the average height is equal to or greater than a minimum height threshold and equal to or less than a maximum height threshold, then determine the average height as the first height and (iii-2) if the average height is less than the minimum height threshold or greater than the maximum height threshold, then determine an average of the minimum height threshold and the maximum height threshold as the first height.

2 more elements

A method for calibrating at least one pitch of at least one camera on a specific vehicle, comprising steps of:

(a) if at least one driving image is acquired from the camera during driving operation of the specific vehicle, a calibration device performing or supporting another device to perform a process of inputting the driving image respectively into an object detection network and a lane detection network, to thereby allow (i) the object detection network to detect one or more objects on the driving image and thus to generate object detection information on the objects, and (ii) the lane detection network to detect one or more lanes on the driving image and thus to generate lane detection information on the lanes;
(b) the calibration device performing or supporting another device to perform (i) a process of profiling the object detection information to thereby generate each piece of object profiling information corresponding to each of the objects and a process of profiling the lane detection information to thereby generate each piece of lane profiling information corresponding to each of the lanes, (ii) a process of inputting the object profiling information into an object-based pitch estimation module to allow the object-based pitch estimation module to (ii-1) select a first target object among the objects by referring to the object profiling information and execute a first pitch estimation using a first height of the first target object, to thereby generate a first pitch and (ii-2) select a second target object among the objects by referring to the object profiling information and execute a second pitch estimation using a width of the second target object, to thereby generate a second pitch, and (iii) a process of inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module, wherein the vanishing point detection information is generated by a VP (vanishing point) estimation network capable of analyzing the driving image and thus detecting at least one vanishing point, to thereby allow the lane-based pitch estimation module to (iii-1) execute a third pitch estimation using the lane profiling information and thus generate a third pitch and (iii-2) execute a fourth pitch estimation using the vanishing point detection information and thus generate a fourth pitch;
and
(c) the calibration device performing or supporting another device to perform a process of inputting the first pitch to the fourth pitch into a pitch-deciding module, to thereby allow the pitch-deciding module to apply ensemble operation to the first pitch to the fourth pitch and thus to generate a decided pitch corresponding to the driving image;
wherein, at the step of (b), the calibration device performs or supports another device to perform a process of instructing the object-based pitch estimation module to select a third specific object, whose lateral distance is smallest, as the second target object, wherein the third specific object is selected among third candidate objects (1) whose lateral distances from the specific vehicle are equal to or less than a third distance threshold, (2) whose bounding boxes are not truncated, (3) whose object classes are a vehicle class, (4) which are not occluded, (5) whose differences between 2D bounding boxes and 3D bounding boxes among the bounding boxes thereof are equal to or less than a box difference threshold, and (6) whose aspect ratios of the bounding boxes thereof are equal to or greater than a ratio threshold, by referring to the object profiling information.

2 more elements