# US 12,217,491 B2 · Method for inspecting labeling on bounding box by using deep learning model and apparatus using same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,491 B2 - Document ID: US12217491B2 - Status: Granted - Kind: B2 - Application number: 17/771,476 - Filed: 2020-12-01 - Published: 2022-11-17 - Granted: 2025-02-04 - Earliest claimed: 2019-12-20 - Assignees: Select Star, Inc. - Applicants: Select Star, Inc. - Inventors: Se Yeob Kim - Web page: https://patentgrep.com/patent/US12217491B2 ## CPC - G06V 10/776: Validation; Performance evaluation - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract According to the present invention, proposed is a method for inspecting a labeling operation, the method comprising, when a deep learning model for inspecting a labeling operation for a bounding box corresponding to an object included in an image is present and a computing apparatus uses the deep learning model, the steps of: performing, by the computing apparatus, first training on the deep learning model on the basis of a training image; obtaining, by the computing apparatus, an operation image and a bounding box labeling value therefor; calculating, by the computing apparatus, a score for inspection by performing a calculation while passing the operation image and the bounding box labeling value through the deep learning model; and determining, by the computing apparatus, whether the bounding box labeling value for the operation image is accurate on the basis of the score for inspection and performing any one of a pass process, a fail process, and a re-inspection process. ## Claims ### Claim 1 A method for inspecting a labeling operation, in which a deep learning model for inspecting the labeling operation for a bounding box corresponding to an object included in an image is present, and a computing apparatus uses the deep learning model, the method comprising: (a) performing, by the computing apparatus, first training on the deep learning model based on a training image; (b) obtaining, by the computing apparatus, an operation image and a bounding box labeling value for the operation image; (c) calculating, by the computing apparatus, an inspection score by performing calculation while passing the operation image and the bounding box labeling value through the deep learning model; and (d) determining, by the computing apparatus, whether the bounding box labeling value for the operation image is accurate based on the inspection score, and performing any one of pass processing, fail processing, and re-inspection processing, wherein, upon a first pair of threshold values provided where a first threshold value of the pair is greater than a second threshold value of the pair, the computing apparatus is configured to: perform the pass processing on the operation image and the bounding box labeling value when the inspection score is greater than or equal to the first threshold value; perform the re-inspection processing on the operation image and the bounding box labeling value when the inspection score is less than the first threshold and greater than or equal to the second threshold value; and perform the fail processing on the operation image and the bounding box labeling value when the inspection score is less than the second threshold value; and wherein, the computing apparatus determines who is qualified to perform the re-inspection processing according to various levels of the labeling operation difficulty by (1) arranging a second pair of threshold values within the first and second threshold values and (2) finding a location of the inspection score within a particular range of the threshold values of the first and second pairs, in which third and fourth threshold values of the second pair are less than the first threshold value and greater than the second threshold value, respectively, and the third threshold value is greater than the fourth threshold value. ### Claim 2 (depends on 1) The method of claim 1, wherein the computing apparatus is configured to transmit the operation image and the bounding box labeling value, which are subject to the re-inspection processing, to an operator terminal so as to support re-inspection to be performed in the operator terminal, and perform second training on the deep learning model based on a result of the re-inspection. ### Claim 3 (depends on 2) The method of claim 2, wherein the computing apparatus is configured to adjust the first threshold value and the second threshold value based on the result of the re-inspection. ### Claim 4 (depends on 1) The method of claim 1, wherein, a plurality of operators are classified based on corresponding operation performance values, when at least a pth threshold value and a qth threshold value that are less than the first threshold value and greater than the second threshold value are present, and the pth threshold value is greater than the qth threshold value, the computing apparatus is configured to: i) support re-inspection to be performed in a terminal of an operator having a low grade when the inspection score is less than the first threshold value and greater than or equal to the pth threshold value, or when the inspection score is less than the qth threshold value and greater than or equal to the second threshold value; and ii) support the re-inspection to be performed in a terminal of an operator having a high grade when the inspection score is less than the pth threshold value and greater than or equal to the qth threshold value. ### Claim 5 (depends on 1) The method of claim 1, wherein the inspection score is a probability value corresponding to a, and satisfies 0<=a<=1. ### Claim 6 (depends on 1) The method of claim 1, wherein, in the step (a), while at least one parameter is present to perform the calculation of the deep learning model, the computing apparatus is configured to: i) adjust the at least one parameter so that a result of performing the calculation while passing the training image and a correct answer bounding box labeling value through the deep learning model is 1; and ii) adjust the at least one parameter so that a result of performing the calculation while passing the training image and an incorrect answer bounding box labeling value through the deep learning model is 0. ### Claim 7 A computing apparatus, which is an apparatus for inspecting a labeling operation, in which a deep learning model for inspecting the labeling operation for a bounding box corresponding to an object included in an image is present, and the computing apparatus uses the deep learning model, the computing apparatus comprising: a communication unit for transmitting and receiving information to and from an operation terminal; and a processor for performing first training on the deep learning model based on a training image, obtaining an operation image and a bounding box labeling value for the operation image, calculating an inspection score by performing calculation while passing the operation image and the bounding box labeling value through the deep learning model, determining whether the bounding box labeling value for the operation image is accurate based on the inspection score, and performing any one of pass processing, fail processing, and re-inspection processing, wherein, upon a first pair of threshold values provided where a first threshold value of the pair is greater than a second threshold value of the pair, the processor is configured to: perform the pass processing on the operation image and the bounding box labeling value when the inspection score is greater than or equal to the first threshold value; perform the re-inspection processing on the operation image and the bounding box labeling value when the inspection score is less than the first threshold and greater than or equal to the second threshold value; and perform the fail processing on the operation image and the bounding box labeling value when the inspection score is less than the second threshold value; and wherein, the computing apparatus determines who is qualified to perform the re-inspection processing according to various levels of the labeling operation difficulty by (1) arranging a second pair of threshold values within the first and second threshold values and (2) finding a location of the inspection score within a particular range of the threshold values of the first and second pairs, in which third and fourth threshold values of the second pair are less than the first threshold value and greater than the second threshold value, respectively, and the third threshold value is greater than the fourth threshold value.