# US 12,759,151 B2 · Training data generation device and training data generation program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,759,151 B2 - Document ID: US12759151B2 - Status: Granted - Kind: B2 - Application number: 18/725,365 - Filed: 2022-12-16 - Published: 2025-02-27 - Granted: 2026-10-13 - Earliest claimed: 2021-12-28 - Assignees: Kawasaki Jukogyo Kabushiki Kaisha - Applicants: Kawasaki Jukogyo Kabushiki Kaisha - Inventors: Tianfen Xu; Seita Ono; Kippei Matsuda - Web page: https://patentgrep.com/patent/US12759151B2 ## CPC - B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping} - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A controller includes: a label assigner that assigns a label indicating predetermined information to an image object in response to input operation of a user; a generator that generates a group of an image and a label assigned to the image as training data; and a storage that stores a second image on which detection by a detection model was performed and failed for an object, in association with a detection result by a detection model. In the second image, the label assigner assigns the label indicating the predetermined information to a detection failed object in response to input operation of the user and substitutes the label indicating the predetermined information of a detection successful object by the detection result. The generator generates a group of the second image, and the label assigned to the detection failed object and the detection result of the detection successful object as training data. ## Claims ### Claim 1 A training data generator that generates training data for retraining a trained detection model, the training data generator comprising: a processor programmed to: detect an object included in an image; a label to the object included in the image in response to input operation of a user, the label indicating (i) positional information identifying a position of the object within the image, and (ii) attribute information relating to a detection result of the object by the trained detection model; and generate a group of the image and the label assigned to the image as training data; and a storage that stores an image on which detection by the detection model was performed and failed for at least one object, in association with a detection result by the detection model, wherein in the stored image, the processor is programmed to assign the label to a detection failed object in response to input operation of the user and substitutes the label of a detection successful object by the detection result, and the processor is programmed to generate retraining data using the stored image and the positional information included in the label assigned to the detection failed object and such that the retraining data is configured to retrain the trained detection model using the positional information as object position information for detecting the object. ### Claim 2 (depends on 1) The training data generator according to claim 1, further comprising a display that displays the image in the storage, wherein the processor is programmed to assign a mark indicating the positional information and the attribute information to the detection failed object included in the image in the storage displayed by the display in response to input operation of the user to thereby assign the label. ### Claim 3 (depends on 2) The training data generator according to claim 2, wherein the processor is programmed to: cause the display to display the image in the storage, and in causing the display to display the image in the storage, assign the mark indicating the detection result to the detection successful object included in the image in the storage. ### Claim 4 (depends on 2) The training data generator according to claim 2, wherein the processor is programmed to assign the label by using a position of the mark assigned to an object in response to input operation of the user as positional information of the object. ### Claim 5 (depends on 4) The training data generator according to claim 4, wherein the position of the mark is set as a position at which a robot arm performs a predetermined process on the object based on the position of the mark. ### Claim 6 (depends on 5) The training data generator according to claim 5, wherein the position of the mark is set as a position at which the robot arm holds the object. ### Claim 7 (depends on 5) The training data generator according to claim 5, wherein the processor is communicable with a robot controller that controls the robot arm. ### Claim 8 (depends on 2) The training data generator according to claim 2, wherein the mark of a color distinguished based on attribute information of the object is assigned to the object in response to input operation of the user; and the processor is programmed to assign the label by using the color of the mark as attribute information of the object. ### Claim 9 (depends on 1) The training data generator according to claim 1, wherein the detection failed object is an undetected object or a misdetected object. ### Claim 10 A non-transitory storage medium storing a training data generation program that generates training data for retraining a trained detection model, the program causing a computer to perform the functions of: detecting an object included in an image; assigning a label to the object included in the image in response to input operation of a user, the label indicating (i) positional information identifying a position of the object within the image, and (ii) attribute information relating to a detection result of the object by the trained detection model; and generating a group of the image and the label assigned to the image as training data; storing an image on which detection by the detection model was performed and failed for at least one object, in association with a detection result by the detection model; assigning the label to a detection failed object in response to input operation of the user and substituting the label of a detection successful object by the detection result, in the stored image; and generating retraining data using the stored image and the positional information included in the label assigned to the detection failed object and such that the retraining data is configured to retrain the trained detection model using the positional information as object position information for detecting the object. ### Claim 11 A training data generation method for generating training data for retraining a trained detection model, the method comprising: detecting an object included in an image; assigning a label to the object included in the image in response to input operation of a user, the label indicating (i) positional information identifying a position of the object within the image, and (ii) attribute information relating to a detection result of the object by the trained detection model; and generating a group of the image and the label assigned to the image as training data; storing an image on which detection by the detection model was performed and failed for at least one object, in association with a detection result by the detection model; assigning the label to a detection failed object in response to input operation of the user and substituting the label of a detection successful object by the detection result, in the stored image; and generating retraining data using the stored image and the positional information included in the label assigned to the detection failed object and such that the retraining data is configured to retrain the trained detection model using the positional information as object position information for detecting the object.