Robot program generation method from human demonstration

A system for generating a robot motion program from human demonstration of an operation, said system comprising:

• a vision sensor, including a two-dimensional (2D) or three-dimensional (3D) camera;
• a microphone;
• a force sensor;
• and
• a computer having a processor and memory, said computer receiving signals from the vision sensor, the microphone and the force sensor and being configured to generate the robot motion program using steps including;
• defining a state machine model for an operation including a sequence of states and transitions around a continuous loop, where the sequence of states consists of a pick state, a move state, a place state and a depart/approach state;
• analyzing camera images of a hand of a human demonstrator demonstrating the operation on a workpiece to create hand motion data;
• analyzing one or more of camera images of the demonstrating, signals from the force sensor, signals from the microphone and signals from an input signaling device, to create logic data, where the logic data triggers transitions only from a current state to a next state in the state machine model, wherein creating the logic data includes detecting state transitions during the operation, using the state transitions to synchronize gripper activate and deactivate commands with particular time steps in the hand motion data, and including the gripper activate and deactivate commands in the robot motion program, and wherein detecting state transitions includes receiving signals from the force sensor configured to detect forces on the workpiece during the operation and triggering a state transition based on the signals from the force sensor, including detecting a peak followed by a drop-off in the signals from the force sensor to trigger a pick state transition or a place state transition;
• and
• generating the robot motion program based on the hand motion data and the logic data, said program including instructions to cause a robot to manipulate a gripper to perform the operation on the workpiece.

A method for creating a robot motion program from human demonstration, said method comprising:

• defining a state machine model for an operation including a sequence of states and transitions around a continuous loop, where the sequence of states consists of a pick state, a move state, a place state and a depart/approach state;
• demonstrating an operation on a workpiece by a hand of a human demonstrator;
• analyzing camera images of the hand demonstrating the operation on the workpiece, by a computer having a processor and memory, to create hand motion data, wherein creating the hand motion data includes refining the hand motion data before generating the robot motion program, including performing a least squares fit on the hand motion data to create refined hand motion data, and computing a spline interpolation through the refined hand motion data to generate the robot motion program;
• analyzing one or more of camera images of the demonstrating, signals from a force sensor, signals from a microphone and signals from an input signaling device, by the computer, to create logic data, where the logic data triggers transitions only from a current state to a next state in the state machine model;
• and
• generating the robot motion program based on the hand motion data and the logic data, said program including instructions to cause a robot to manipulate a gripper to perform the operation on the workpiece.

A method for creating a robot motion program from human demonstration, said method comprising:

• defining a state machine model for an operation including a sequence of states and transitions around a continuous loop, where the sequence of states consists of a pick state, a move state, a place state and a depart/approach state;
• demonstrating an operation on a workpiece by a hand of a human demonstrator;
• analyzing camera images of the hand demonstrating the operation on the workpiece, by a computer having a processor and memory, to create hand motion data, wherein creating the hand motion data includes refining the hand motion data before generating the robot motion program, including replacing the hand motion data with a piecewise linear function having a vertical line segment upward from a start point followed by a horizontal line segment through a highest point in the hand motion data followed by a vertical line segment downward to an end point, and using the piecewise linear function to generate the robot motion program;
• analyzing one or more of camera images of the demonstrating, signals from a force sensor, signals from a microphone and signals from an input signaling device, by the computer, to create logic data, where the logic data triggers transitions only from a current state to a next state in the state machine model;
• and
• generating the robot motion program based on the hand motion data and the logic data, said program including instructions to cause a robot to manipulate a gripper to perform the operation on the workpiece.