Object-centric prediction and control for human to robot skill transfer

A system for performing a task with an object, comprising:

• a robotic hand to perform a task with an object in a robotic environment, the robotic hand including digits having digit sections including a digit having a fingertip;
• a sensor array coupled to the robotic hand, the sensor array including i) tactile sensors arranged in sensor sections coupled to digit sections, including a sensor section coupled to the fingertip, the sensor section including a plurality of tactile sensors, each tactile sensor including circuitry to generate a digital output, and ii) motion sensors coupled to digits;
• a camera to generate images of the object;
• and
• a processor configured to:
• receive current contact measurements from the sensor array, including tactile signals determined from tactile sensors and positions determined from motion sensors, the tactile signals including digital outputs from tactile sensors indicating an applied force;
• receive a current pose of the object based on an image from the camera, the current pose including an orientation of the object in the robotic environment;
• and
• generate, based on the current contact measurements from the sensor array, and the current pose of the object from the image, corresponding to one another at a time stamp, a prediction from a machine learning model of:
• future contact measurements of the sensor array, and a future pose of the object, corresponding to one another at a future time stamp, to control the robotic hand to perform the task with the object.

A method for controlling a robotic device to perform a task, comprising:

• receiving current contact measurements from a sensor array coupled to a robotic hand in a robotic environment, the robotic hand including digits having digit sections, including a fingertip, the sensor array including i) tactile sensors arranged in sensor sections coupled to digit sections, including a sensor section coupled to the fingertip, the sensor section including a plurality of tactile sensors, each tactile sensor including circuitry to generate a digital output, and ii) motion sensors coupled to digits, the current contact measurements including:
• tactile signals determined from tactile sensors and positions determined from motion sensors, the tactile signals including digital outputs from tactile sensors indicating an applied force;
• receiving a current pose of an object based on one or more images from a camera, the current pose including an orientation of the object in the robotic environment;
• generating, based on the current contact measurements from the sensor array, and the current pose of the object from the one or more images, corresponding to one another at a time stamp, a prediction from a machine learning model, the prediction including:
• future contact measurements of the sensor array, and a future pose of the object, corresponding to one another at a future time stamp, to control the robotic hand to perform a task with the object.