An auto labeling device for generating training data to be used for training a deep learning network capable of analyzing images, comprising:
- at least one memory that stores instructions;
- and
- at least one processor configured to execute the instructions to perform or support another device to perform:
- (I) if one or more test images are acquired, (i) a process of allowing a labeling network to label each of the test images and thus to generate each of labeled test images including primary labeling information and primary confidence scores, on one or more primary objects in each of the test images wherein each of the primary confidence scores is a value ranging from 0 to 1 representing a probability that each of the primary objects is present in its corresponding each of primary labeled regions included in the primary labeling information, (ii) a process of allowing at least one labeler to verify each piece of the primary labeling information for each of the labeled test images, to thereby generate correction-related class information, wherein the correction-related class information includes information on a first verification class representing a class of a first primary label, among primary labels of the primary labeling information, required to be corrected or having been corrected and a second verification class representing a class of a second primary label, among the primary labels of the primary labeling information, not required to be corrected or not having been corrected, for each piece of the primary labeling information, (iii) a process of setting a first threshold confidence score to be used for determining whether it becomes unnecessary for the labeler to verify the primary labeling information by referring to the primary confidence scores and the correction-related class information, respectively corresponding to the primary labeling information, and (iv) a process of setting a second threshold confidence score as a preset maximal allowed error, wherein the preset maximal allowed error is less than the first threshold confidence score, (II) if one or more unlabeled images are acquired, (i) a process of allowing the labeling network to label each of the unlabeled images and thus to generate labeled images including secondary labeling information and secondary confidence scores, on one or more secondary objects in each of the unlabeled images wherein each of the secondary confidence scores is a value ranging from 0 to 1 representing a probability that each of the secondary objects is present in its corresponding each of secondary labeled regions included in the secondary labeling information, (ii) a process of allowing an object difficulty estimation module to (ii-1) generate each of object difficulty scores on each piece of the secondary labeling information by using (1) each of the secondary confidence scores corresponding to each piece of the secondary labeling information and (2) the first threshold confidence score, wherein each of the object difficulty scores is a value ranging from 0 to 1 representing a probability of necessity for verifying each piece of the secondary labeling information, and (ii-2) determine each of object difficulty classes on each piece of the secondary labeling information by using (1) each of the secondary confidence scores corresponding to each piece of the secondary labeling information and (2) the second threshold confidence score, wherein the object difficulty classes include a hard object class representing a first part, among the secondary labeling information, required to be verified by the labeler and an easy object class representing a second part, among the secondary labeling information, not required to be verified by the labeler, (iii) a process of allowing an image difficulty estimation module to (iii-1) generate each of image difficulty scores on each of the labeled images by referring to each of the object difficulty scores on each piece of the secondary labeling information, wherein each of the image difficulty scores is a value ranging from 0 to 1 representing a probability of necessity for verifying each of the labeled images, and (iii-2) determine each of image difficulty classes on each of the labeled images by referring to the image difficulty scores, wherein the image difficulty classes include a hard image class representing a class of one or more first labeled images, among the labeled images, required to be verified by the labeler and an easy image class representing a class of one or more second labeled images, among the labeled images, not required to be verified by the labeler, and (III) (i) a process of transmitting, to the labeler, the first labeled images which are labeled as the hard image class by referring to each of the image difficulty classes on each of the labeled images, to thereby allow the labeler to verify the first labeled images and thus to generate verified labeled images, and (ii) a process of generating the training data comprised of (1) the second labeled images which are labeled as the easy image class and (2) the verified labeled images.