Method and device for on-vehicle active learning to be used for training perception network of autonomous vehicle

A method for on-vehicle active learning to be used for training a perception network of an autonomous vehicle, comprising steps of:

• (a) an on-vehicle active learning device, if a driving video and sensing information are acquired respectively from a camera and one or more sensors mounted on an autonomous vehicle while the autonomous vehicle is driven, performing or supporting another device to perform a process of inputting one or more consecutive frames of the driving video and the sensing information into a scene code assigning module, to thereby allow the scene code assigning module to generate each of one or more scene codes including information on each of scenes in each of the frames and information on one or more driving events by referring to the frames and the sensing information;
• and
• (b) the on-vehicle active learning device performing or supporting another device to perform at least one of (i) a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle by using each of the scene codes of each of the frames and the object detection information, for each of the frames, detected by an object detector and a process of storing the specific frames and their corresponding specific scene codes in a frame storing part such that the specific frames and their corresponding specific scene codes match with one another and (ii) a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames by using the scene codes and the object detection information and a process of storing the specific frames and their corresponding specific scene codes in the frame storing part such that the specific frames and their corresponding specific scene codes match with one another;
• and
• wherein, at the step of (a), the on-vehicle active learning device performs or supports another device to perform a process of allowing the scene code assigning module to (i) apply a learning operation to each of the frames, to thereby classify each of the scenes of each of the frames into one of classes of driving environments and one of classes of driving roads and thus generate each of class codes of each of the frames, via a scene classifier based on deep learning, (ii) detect each of driving events, which occurs while the autonomous vehicle is driven, by referring to each of the frames and each piece of the sensing information on each of the frames, to thereby generate each of event codes, via a driving event detecting module, and (iii) generate each of the scene codes for each of the frames by using each of the class codes of each of the frames and each of the event codes of each of the frames.

A method for on-vehicle active learning to be used for training a perception network of an autonomous vehicle, comprising steps of:

• (a) an on-vehicle active learning device, if a driving video and sensing information are acquired respectively from a camera and one or more sensors mounted on an autonomous vehicle while the autonomous vehicle is driven, performing or supporting another device to perform a process of inputting one or more consecutive frames of the driving video and the sensing information into a scene code assigning module, to thereby allow the scene code assigning module to generate each of one or more scene codes including information on each of scenes in each of the frames and information on one or more driving events by referring to the frames and the sensing information;
• and
• (b) the on-vehicle active learning device performing or supporting another device to perform at least one of (i) a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle by using each of the scene codes of each of the frames and the object detection information, for each of the frames, detected by an object detector and a process of storing the specific frames and their corresponding specific scene codes in a frame storing part such that the specific frames and their corresponding specific scene codes match with one another and (ii) a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames by using the scene codes and the object detection information and a process of storing the specific frames and their corresponding specific scene codes in the frame storing part such that the specific frames and their corresponding specific scene codes match with one another;
• and
• wherein, at the step of (b), the on-vehicle active learning device performs or supports another device to perform a process of selecting a certain frame, on which an object is detected from its collision area, corresponding to a normal event, as one of the specific frames by referring to the scene codes, wherein the collision area is an area, in the certain frame, where an object is estimated as being located if the autonomous vehicle collides with the object or where the object is estimated to be located if the autonomous vehicle is estimated to collide with the object.

An on-vehicle active learning device for on-vehicle active learning to be used for training a perception network of an autonomous vehicle, comprising:

• at least one memory that stores instructions;
• and
• at least one processor configured to execute the instructions to perform or support another device to perform:
• (I) if a driving video and sensing information are acquired respectively from a camera and one or more sensors mounted on an autonomous vehicle while the autonomous vehicle is driven, a process of inputting one or more consecutive frames of the driving video and the sensing information into a scene code assigning module, to thereby allow the scene code assigning module to generate each of one or more scene codes including information on each of scenes in each of the frames and information on one or more driving events by referring to the frames and the sensing information and (II) at least one of (i) a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle by using each of the scene codes of each of the frames and the object detection information, for each of the frames, detected by an object detector and a process of storing the specific frames and their corresponding specific scene codes in a frame storing part such that the specific frames and their corresponding specific scene codes match with one another and (ii) a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames by using the scene codes and the object detection information and a process of storing the specific frames and their corresponding specific scene codes in the frame storing part such that the specific frames and their corresponding specific scene codes match with one another;
• and
• wherein the processor further performs or supports another device to perform:
• (III) (III-1) a process of sampling the specific frames stored in the frame storing part by using the specific scene codes to thereby generate training data and (III-2) a process of executing on-vehicle learning of the perception network of the autonomous vehicle by using the training data.

An on-vehicle active learning device for on-vehicle active learning to be used for training a perception network of an autonomous vehicle, comprising:

• at least one memory that stores instructions;
• and
• at least one processor configured to execute the instructions to perform or support another device to perform:
• (I) if a driving video and sensing information are acquired respectively from a camera and one or more sensors mounted on an autonomous vehicle while the autonomous vehicle is driven, a process of inputting one or more consecutive frames of the driving video and the sensing information into a scene code assigning module, to thereby allow the scene code assigning module to generate each of one or more scene codes including information on each of scenes in each of the frames and information on one or more driving events by referring to the Response to Non-Final Office Action frames and the sensing information and (II) at least one of (i) a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle by using each of the scene codes of each of the frames and the object detection information, for each of the frames, detected by an object detector and a process of storing the specific frames and their corresponding specific scene codes in a frame storing part such that the specific frames and their corresponding specific scene codes match with one another and (ii) a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames by using the scene codes and the object detection information and a process of storing the specific frames and their corresponding specific scene codes in the frame storing part such that the specific frames and their corresponding specific scene codes match with one another;
• and
• wherein, at the process of (I), the processor performs or supports another device to perform a process of allowing the scene code assigning module to (i) apply a learning operation to each of the frames, to thereby classify each of the scenes of each of the frames into one of classes of driving environments and one of classes of driving roads and thus generate each of class codes of each of the frames, via a scene classifier based on deep learning, (ii) detect each of driving events, which occurs while the autonomous vehicle is driven, by referring to each of the frames and each piece of the sensing information on each of the frames, to thereby generate each of event codes, via a driving event detecting module, and (iii) generate each of the scene codes for each of the frames by using each of the class codes of each of the frames and each of the event codes of each of the frames.

An on-vehicle active learning device for on-vehicle active learning to be used for training a perception network of an autonomous vehicle, comprising:

• at least one memory that stores instructions;
• and
• at least one processor configured to execute the instructions to perform or support another device to perform:
• (I) if a driving video and sensing information are acquired respectively from a camera and one or more sensors mounted on an autonomous vehicle while the autonomous vehicle is driven, a process of inputting one or more consecutive frames of the driving video and the sensing information into a scene code assigning module, to thereby allow the scene code assigning module to generate each of one or more scene codes including information on each of scenes in each of the frames and information on one or more driving events by referring to the frames and the sensing information and (II) at least one of (i) a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle by using each of the scene codes of each of the frames and the object detection information, for each of the frames, detected by an object detector and a process of storing the specific frames and their corresponding specific scene codes in a frame storing part such that the specific frames and their corresponding specific scene codes match with one another and (ii) a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames by using the scene codes and the object detection information and a process of storing the specific frames and their corresponding specific scene codes in the frame storing part such that the specific frames and their corresponding specific scene codes match with one another;
• and
• wherein, at the process of (II), the processor performs or supports another device to perform a process of selecting a certain frame, on which an object is detected from its collision area, corresponding to a normal event, as one of the specific frames by referring to the scene codes, wherein the collision area is an area, in the certain frame, where an object is estimated as being located if the autonomous vehicle collides with the object or where the object is estimated to be located if the autonomous vehicle is estimated to collide with the object.