Vision-based teleoperation of dexterous robotic system
AssigneeNVIDIA Corporation
InventorsAnkur Handa, Karl Van Wyk, Wei Yang, Yu-Wei Chao, Dieter Fox, Qian Wan
Life of this patent
Estimating the term…
Abstract
A human pilot controls a robotic arm and gripper by simulating a set of desired motions with the human hand. In at least one embodiment, one or more images of the pilot's hand are captured and analyzed to determine a set of hand poses. In at least one embodiment, the set of hand poses is translated to a corresponding set of robotic-gripper poses. In at least one embodiment, a set of motions is determined that perform the set of robotic-gripper poses, and the robot is directed to perform the set of motions.
Claims31
A computer-implemented method comprising:
- determining, from an image of a hand, at least one first joint angle of a first joint of at least one digit of a model of the hand;
- determining a distance and a direction from the at least one digit to another digit of the model of the hand;
- translating the at least one first joint angle of the first joint to at least one second joint angle of a corresponding second joint of a robotic hand based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between at least a pair of robotic digits of the robotic hand;
- determining a set of movements that repositions the robotic hand from a first pose to a second pose based at least in part on the at least one second joint angle of the corresponding second joint;
- and
- causing the robotic hand to perform the set of movements.
A system comprising:
- one or more processors;
- and
- computer-readable memory storing executable instructions that, as a result of being executed by the one or more processors, cause the system to:
- determine, from an image of an appendage performing a task, at least one first joint angle of a first joint of at least one digit of the appendage;
- determine a distance and a direction from the at least one digit to another digit of the appendage;
- determine at least one second joint angle of a second joint of a robotic gripper based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between individually movable portions of the robotic gripper;
- determine a set of movements that repositions the robotic gripper from a first pose to a second pose based at least in part on the at least one second joint angle of the second joint of the robotic gripper;
- and
- perform the set of movements to position the robotic gripper in the second pose to cause the robotic gripper to perform the task.
Non-transitory computer-readable media storing instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:
- determine, from an image of a human performing a task, at least one first joint angle of a joint of at least one digit of a model of an appendage of the human;
- determine a distance and a direction from the at least one digit to another digit of the model of the appendage;
- translate the at least one first joint angle of the first joint to at least one second joint angle of a second joint of a robotic hand based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between at least a pair of robotic digits of the robotic hand;
- determine a set of movements that repositions the robotic hand from a first pose to a second pose based at least in part on the at least one second joint angle of the second joint of the robotic hand;
- and
- cause the robotic hand to perform the set of movements.
Dependencies are inferred from the claim text.