Back to results
US 10,919,152 B1Granted
Teleoperating of robots with tasks by mapping to human operator pose
AssigneeNimble Robotics, Inc.
InventorSimon Kalouche
Life of this patent
Estimating the term…
Abstract
A system enables teleoperation of a robot based on a pose of a subject. The system includes an image capturing device and an operator system controller that are remotely located from a robotic system controller and a robot. The image capturing device captures images of the subject. The operator system controller maps a processed version of the captured image to a three-dimensional skeleton model of the subject and generates body pose information of the subject in the captured image. The robotic system controller communicates with the operator system controller over a network. The robotic system controller generates a plurality of kinematic parameters for the robot and causes the robot to take a pose corresponding to the pose of the subject in the captured image.
Claims25
A system comprising:
- an imitation learning engine;
- an operator system controller coupled to an image capturing device, the operator system controller configured to:
- process an image of a subject, captured by the image capturing device, using a machine learning algorithm to identify one or more body parts of the subject during execution of a task;
- and
- generate, based on the one or more body parts of the subject, body pose information of the subject in the captured image, the body pose information indicating a pose or motion trajectory of the subject in the captured image;
- and
- a robotic system controller communicating with the operator system controller over a network, the robotic system controller coupled to a second image capturing device, the robotic system controller configured to:
- receive one or more images of a robot and/or an environment surrounding the robot, captured by the second image capturing device, during execution of the task;
- generate one or more pose and/or motion commands by processing the body pose information received from the operator system controller;
- control one or more actuators of the robot according to the one or more pose and/or motion commands to cause the robot to take a pose or motion trajectory corresponding to the pose or motion trajectory of the subject in the captured image;
- and
- provide the one or more images of the robot and/or the environment surrounding the robot, the one or more pose and/or motion commands, and/or information identifying the task to the imitation learning engine.
9 more elements
A method of operating a robot comprising:
- by an operator system controller:
- processing an image of a subject using a machine learning algorithm to identify one or more body parts of the subject during execution of a task;
- generating, based on the one or more body parts of the subject, body pose information of the subject in the captured image, the body pose information indicating a pose or motion trajectory of the subject in the image;
- and
- by a robotic system controller:
- receiving one or more images of the robot and/or an environment surrounding the robot, captured by a second image capturing device, during execution of the task;
- generating one or more pose and/or motion commands by processing the body pose information received from the operator system controller;
- controlling one or more actuators of the robot according to the one or more pose and/or motion commands to cause the robot to take a pose or motion trajectory corresponding to the pose or motion trajectory of the subject in the image and
- providing the one or more images of the robot and/or the environment surrounding the robot, the one or more pose and/or motion commands, and/or information identifying the task to the imitation learning engine.
6 more elements
A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- processing an image of a subject using a machine learning algorithm to identify one or more body parts of the subject during execution of a task;
- generating, based on the identified one or more body parts of the subject, body pose information of the subject in the image, the body pose information indicating a pose or motion trajectory of the subject in the image;
- receiving one or more images of a robot and/or an environment surrounding the robot during execution of the task;
- generating one or more pose and/or motion commands by processing the body pose information;
- controlling one or more actuators of the robot according to the one or more pose and/or motion commands to cause the robot to take a pose or motion trajectory corresponding to the pose or motion trajectory of the subject in the image;
- and
- providing the one or more images of the robot and/or the environment surrounding the robot, the one or more pose and/or motion commands, and/or information identifying the task to an imitation learning engine.
4 more elements
A system comprising:
- an imitation learning engine
- an operator system controller coupled to an image capturing device, the operator system controller configured to:
- generate body pose information of a subject captured in an image by the image capturing device, the body pose information indicating a pose or motion trajectory of the subject in the captured image;
- and
- a robotic system controller communicating with the operator system controller over a network, the robotic system controller configured to:
- generate one or more pose and/or motion commands by processing the body pose information received from the operator system controller;
- control one or more actuators of the robot according to the one or more pose and/or motion commands to cause the robot to take a pose or motion trajectory corresponding to the pose or motion trajectory of the subject in the captured image;
- and
- provide the one or more pose and/or motion commands to the imitation learning engine.
6 more elements
Dependencies are inferred from the claim text.