Dual arm robot teaching from dual hand human demonstration

A method for programming a dual arm robot system to perform an operation by dual hand human demonstration, said method comprising:

• demonstrating the operation on workpieces by a human using both hands;
• analyzing camera images of the hands demonstrating the operation on the workpieces, by a computer, to create demonstration data including gripper poses computed from three-dimensional (3D) coordinates of key points of the hands, where the 3D coordinates of the key points are determined from the images by a first neural network used to identify left and right hands in the images and a second neural network used to compute the 3D coordinates in sub-images of the identified left and right hands;
• refining the demonstration data, including performing object localization using edge detection to compensate action points, replacing the demonstration data with simplified versions, or both;
• generating robot motion commands, based on the demonstration data, to cause the dual arm robot system to perform the operation on the workpieces, where one robot arm performs tasks demonstrated by one of the human's hands and the other robot arm performs tasks demonstrated by the other of the human's hands;
• and
• performing the operation on the workpieces by the dual arm robot system.

A system for dual arm robot teaching from dual hand human demonstration, said system comprising:

• a three-dimensional (3D) camera;
• a computer having a processor and memory, said computer being configured to perform steps including;
• receiving a sequence of images from the 3D camera, the images including a left hand and a right hand of a human demonstrating an operation on one or more workpieces,
• analyzing each of the images using a first neural network to determine an identity and a location of the left hand and the right hand in the image,
• analyzing left hand and right hand sub-images using a second neural network to determine coordinates of a plurality of key points on the left and right hands,
• applying pixel depth data from the 3D camera to the key points to produce 3D left and right hand pose data for each of the images,
• combining the 3D left and right hand pose data for the sequence of images into 3D left and right hand motion traces defining the operation, and
• refining the 3D left and right hand motion traces, including performing object localization using edge detection, replacing the motion traces with simplified versions, or both;
• and
• a robot controller in communication with the computer, said controller using the refined 3D left and right hand motion traces by to control a dual arm robot system to perform the operation, where the left hand motion trace is performed by one robot arm and the right hand motion trace is performed by the other robot arm.