US 12,172,303 B2 · Robot teaching by demonstration with visual servoing

Abstract

A method for teaching and controlling a robot to perform an operation based on human demonstration with images from a camera. The method includes a demonstration phase where a camera detects a human hand grasping and moving a workpiece to define a rough trajectory of the robotic movement of the workpiece. Line features or other geometric features on the workpiece collected during the demonstration phase are used in an image-based visual servoing (IBVS) approach which refines a final placement position of the workpiece, where the IBVS control takes over the workpiece placement during the final approach by the robot. Moving object detection is used for automatically localizing both object and hand position in 2D image space, and then identifying line features on the workpiece by removing line features belonging to the hand using hand keypoint detection.

Claims20

Dependent claims

A method for programming a robot to perform an operation by human demonstration, said method comprising:

demonstrating the operation on a workpiece by a human hand;
analyzing camera images of the hand and the workpiece, by a computer, to create demonstration data;
adding a new point to a trajectory at each image frame interval based on a hand coordinate frame identified in the demonstration data;
identifying geometric features on the workpiece in the demonstration data at each image frame interval;
finalizing the trajectory and capturing target locations of the geometric features when the demonstration of the operation is completed;
executing a program, by a robot controller, to cause a robot with a gripper to perform the operation on the workpiece using the trajectory;
starting image-based visual servoing (IBVS) when it is determined that a final approach position has been reached in the trajectory;
and
using IBVS control, by the robot controller, to maneuver the gripper until current locations of the geometric features on the workpiece match the target locations of the geometric features.

6 more elements

A system for programming a robot to perform an operation by human demonstration, said system comprising:

a camera;
a robot with a gripper;
and
a robot controller having a processor and memory, said controller being in communication with the robot and receiving images from the camera, said controller being configured to perform steps including;
analyzing camera images of a human hand demonstrating the operation on the workpiece to create demonstration data;
adding a new point to a trajectory at each image frame interval based on a hand coordinate frame identified in the demonstration data;
identifying geometric features on the workpiece in the demonstration data at each image frame interval;
finalizing the trajectory and capturing target locations of the geometric features when the demonstration of the operation is completed;
executing a program to cause the robot with the gripper to perform the operation on the workpiece using the trajectory;
starting image-based visual servoing (IBVS) when it is determined that a final approach position has been reached in the trajectory;
and
using IBVS control to maneuver the gripper until current locations of the geometric features on the workpiece match the target locations of the geometric features.

9 more elements

Dependencies are inferred from the claim text.

Robot teaching by demonstration with visual servoing

Life of this patent

Robot teaching by demonstration with visual servoing

Life of this patent

Abstract

Claims20