Human skill learning by inverse reinforcement learning

A method for teaching a robot to perform an operation by human demonstration, said method comprising:

• performing the demonstration of the operation by a human hand, including installing a moving workpiece into an aperture in a fixed workpiece;
• recording force and motion data from the demonstration, by a computer, to create demonstration data, including demonstration state data and demonstration action data, where the demonstration data includes translational and rotational velocities of the moving workpiece and contact forces and torques between the moving workpiece and the fixed workpiece;
• using the demonstration data to train a first neural network to output a first distribution of probabilities associated with the demonstration state data and the demonstration action data;
• performing the operation by a robot, including using a robot controller configured with a policy neural network which determines robot action data to provide as a robot motion command based on robot state data provided as feedback from the robot;
• computing a value of a reward function following completion of the operation by the robot, including using the first neural network to output a second distribution of probabilities associated with the robot state data and the robot action data, and using the first and second distributions of probabilities in a Kullback-Leibler (KL) divergence calculation in the reward function, where the reward function includes a KL divergence term and a success term;
• and
• using the value of the reward function in ongoing reinforcement learning training of the policy neural network to maximize the value of the reward function.

A system for teaching a robot to perform an operation by human demonstration, said system comprising:

• a demonstration workcell including a three-dimensional (3D) camera and a force sensor providing data to a computer, where a human uses a hand to perform the demonstration of the operation by manipulating a moving workpiece relative to a fixed workpiece;
• and
• a robot workcell including a robot in communication with a controller,
• where the computer is configured to record force and motion data from the demonstration to create demonstration data, including demonstration state data and demonstration action data, and use the demonstration data to train a first neural network to output a first distribution of probabilities associated with the demonstration state data and the demonstration action data,
• and where the controller is configured with a policy neural network which determines robot action data to provide as a robot motion command based on robot state data provided as feedback from the robot,
• and the computer or the controller is configured to compute a value of a reward function following completion of the operation by the robot, including using the first neural network to output a second distribution of probabilities associated with the robot state data and the robot action data, use the first and second distributions of probabilities in a Kullback-Leibler (KL) divergence calculation in the reward function, and use the value of the reward function in ongoing reinforcement learning training of the policy neural network.