# US 12,005,592 B2 · Creating training data variability in machine learning for object labelling from images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,005,592 B2 - Document ID: US12005592B2 - Status: Granted - Kind: B2 - Application number: 17/603,175 - Filed: 2020-03-21 - Published: 2022-06-30 - Granted: 2024-06-11 - Earliest claimed: 2019-04-18 - Assignees: Alma Mater Studiorum - Università di Bologna - Applicants: Alma Mater Studiorum Universita' Di Bologna - Inventors: Daniele De Gregorio; Luigi Di Stefano - Web page: https://patentgrep.com/patent/US12005592B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/1612: {characterised by the hand, wrist, grip control} - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/74: {involving reference images or patches} - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G05B 2219/40564: Recognize shape, contour of object, extract position and orientation - G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local - G06N 20/00: Machine learning - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30164: Workpiece; Machine component - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract It is described an image labeling system (100) comprising: a support (2) for an object (3) to be labeled; a digital camera (1) configured to capture a plurality of images of a scene including said object (3); a process and control apparatus (5) configured to receive said images and generate corresponding labeling data (21-24, L1-L4) associated to said object (3); a digital display (4) associated with said support (2) and connected to the process and control apparatus (5) to selectively display additional images (7-13) selected from the group comprising: first images (7-11) in the form of backgrounds for the plurality of images and introducing a degree of variability in the scene; second images (12) indicating position and/or orientation according to which place said object (3) by a user on the support (2); third images (13) to be captured by the digital camera (1) and provided to the process and control apparatus (5) to evaluate a position of the digital camera (1) with respect the digital display (4); fourth images to be captured by the digital camera (1) and provided to the process and control apparatus (5) to evaluate at least one of the following data of the object (3): position, orientation, 3D shape. ## Claims ### Claim 1 Image labeling system (100) comprising: a support (2) for an object (3) to be labeled; a digital camera (1) configured to capture a plurality of images of a scene including said object (3); a process and control apparatus (5) configured to receive said plurality of images and generate corresponding labeling data (21-24, L1-L4) associated to said object (3); a digital display (4) associated with said support (2) and connected to the process and control apparatus (5) to selectively display additional images (7-13) selected from the group comprising: first images (7-11) in the form of backgrounds for the plurality of images and introducing a degree of variability in the scene to be captured by the digital camera (1) and processed by the process and control apparatus (5); second images (12) indicating position and/or orientation according to which place said object (3) by a user on the support (2) in order to be captured by the digital camera (1); third images (13) to be captured by the digital camera (1) and provided to the process and control apparatus (5) to evaluate a position of the digital camera (1) with respect the digital display (4); fourth images to be produced when the object (3) is already placed on the support (2), captured by the digital camera (1) and provided to the process and control apparatus (5) to evaluate at least one of the following data of the object (3): position, orientation, 3D shape. ### Claim 2 (depends on 1) System (100) according to claim 1, wherein the process and control apparatus (5) is configured to cause the digital display (4) to selectively display the first images (7-11) comprising a first background image (7) having corresponding colour and texture and a second background image (10) having relative colour and/or texture different from said colour and texture of the first background image. ### Claim 3 (depends on 1) System (100) according to claim 1, wherein: said digital camera (1) is movable with respect to said support (2); the process and control apparatus (5) is configured to store the following data: optical parameters of said digital camera (1), position data and orientation data of said digital camera (1), position data and orientation data of said object (3). ### Claim 4 (depends on 1) System (100) according to claim 1, wherein said corresponding labeling data are selected from the following data group: text description of the scene represented in the captured image, 2D Bounding Contour (21, 22, 24), 3D Bounding Shape, 3D Pose information (23), Depth Map, Semantic Map (L1-L4). ### Claim 5 (depends on 2) System according to claim 2, further provided with a movable support (6) for supporting and displacing the digital camera (1) or the digital display (4) to capture said scene according to different point of views. ### Claim 6 (depends on 5) System according to claim 5, wherein the movable support is a mechanical manipulator (6) selected from the group: cartesian/gantry robot, cylindrical robot, spherical robot, SCARA robot, articulated robot, parallel robot. ### Claim 7 (depends on 5) System according to claim 5, wherein the movable support is a passive arm. ### Claim 8 (depends on 2) System according to claim 2, wherein digital camera (1) is hand supported and comprises a plurality of sensors to provide said position and orientation data of the digital camera (1) to the process and control apparatus (5). ### Claim 9 (depends on 1) System according to claim 1, wherein the process and control apparatus (5) is configured to display the third images (13) with a pre-established frequency to avoid that the third images (13) appear in said plurality of images of a scene including said object (3). ### Claim 10 (depends on 6) System according to claim 6, wherein the mechanical manipulator (6) further comprises a gripper (14) configured to grab the object (3).