# US 11,897,142 B2 · Method and device for creating a robot control program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,897,142 B2 - Document ID: US11897142B2 - Status: Granted - Kind: B2 - Application number: 17/283,028 - Filed: 2019-10-04 - Published: 2021-11-04 - Granted: 2024-02-13 - Earliest claimed: 2018-10-06 - Assignees: Bystronic Laser AG - Applicants: Bystronic Laser AG - Inventors: Friedemann Seibert - Web page: https://patentgrep.com/patent/US11897142B2 ## CPC - B25J 9/1656: {characterised by programming, planning systems for manipulators} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1697: {Vision controlled systems} - G05B 19/42: Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - G05B 2219/40116: Learn by operator observation, symbiosis, show, watch - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract The present disclosure relates to a method for creating a robot control program for operating a machine tool, in particular a bending machine, having the steps of generating image material of a machining operation of a workpiece on the machine tool by means of at least one optical sensor; extracting at least one part of the workpiece and/or at least one part of a hand of an operator handling the workpiece from the image material; generating a trajectory and/or a sequence of movement points of at least one part of the workpiece and/or at least one part of a hand of an operator from the extracted image material; and creating a robot control program by reverse transformation of the trajectory and/or the sequence of movement points. ## Claims ### Claim 1 A method for creating a robot control program for operating bending tool, the method comprising the steps of: generating image material of a machining operation of a workpiece on the bending machine by means of at least one optical sensor; extracting at least one part of at least one of the workpiece and at least one part of a hand of an operator handling the workpiece from the image material by comparing a 3D object of the workpiece known from a bending plan with the image material; generating at least one of a trajectory and a sequence of movement points of at least one of at least one part of the workpiece and at least one part of a hand of an operator from the extracted image material; creating a robot control program by reverse transformation of at least one of the trajectory and the sequence of movement points; dividing the image material into handling sections, in which the hand is in contact with the workpiece, and into machining sections, in which the hand is not in contact with the workpiece; and controlling gripping tools of the robot accordingly at the transitions between the sections. ### Claim 2 (depends on 1) The method according to claim 1, wherein during extraction of the spatial position of the workpiece, at least one of at least the spatial position of a gripping region of the workpiece and the spatial position of the hand is determined. ### Claim 3 (depends on 1) The method according to claim 1, wherein at least one of the trajectory and the sequence of movement points is present in machine coordinates and that, before the reverse transformation, a transformation into robot coordinates occurs. ### Claim 4 (depends on 1) The method according to claim 1, wherein finger positions of the operator are detected and used to control gripping tools of the robot. ### Claim 5 (depends on 1) The method according to claim 1, wherein, after completion of at least one of a machining step and a bending operation, the shape of the 3D object is updated accordingly. ### Claim 6 (depends on 1) The method according to claim 1, wherein at least one of the trajectory and the sequence of movement points are saved, and that at least one of the saved trajectory and sequence of movement points is transmitted to at least one robot controller for creating a robot control program. ### Claim 7 (depends on 1) The method according to claim 1, wherein the image material is generated by means of a camera and that a spatial position of the workplace is determined on the basis of at least two optical features, the at least two optical features comprising at least one of edges, corners and surfaces. ### Claim 8 (depends on 1) The method according to claim 1, wherein the image material is generated by means of two cameras and that a spatial position of the workpiece is determined on the basis of at least one optical feature, the at least one optical feature comprising at least one of-an edge, corner or surface. ### Claim 9 A device for creating a robot control program for operating a bending machine, the device comprising: the bending machine configured to machine a workpiece; at least one optical sensor configured to generate image material of a machining operation of the workpiece; and at least one computing unit configured for: extracting at least one of at least one part of the workpiece and at least one part of a hand of an operator handling the workpiece from the image material by comparing a 3D object of the workplace known from a bending plan with the image material, generating at least one of a trajectory and a sequence of movement points from the extracted image material, creating a robot control program by reverse transformation of at least one of the trajectory and the sequence of movement points, and wherein the computing unit is configured for dividing the image material into handling sections, in which the hand is in contact with the workpiece, and into machining sections, in which the hand is not in contact with the workpiece, and wherein gripping tools of the robot are controlled accordingly at the transitions between the sections. ### Claim 10 (depends on 9) The device according to claim 9, wherein: the computing unit is connected to the bending machine, the optical sensor is configured for at least one of extracting at least one part of the workpiece and at least one part of a hand of an operator handling the workpiece, from the image material and for generating at least one of a trajectory and a sequence of movement points from the extracted image material, and a robot controller is configured for creating a robot control program by reverse transformation of at least one of the trajectory the sequence of movement points. ### Claim 11 (depends on 9) The device according to claim 9 wherein the computing unit is configured for updating, after completion of at least one of a machining step and a bending operation, the shape of the 3D object, accordingly. ### Claim 12 (depends on 9) The device according to claim 9, further comprising a camera configured for generating the image material, and the computing unit is configured for determining a spatial position of the workpiece on the basis of at least two optical features, the at least two optical features comprising at least one of edges, corners, and surfaces.