Universal manipulation interface

A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:

• collect sensor data about a robotic manipulator within an environment;
• pre-process the sensor data to compensate for an observation latency of at least one sensor associated with the robotic manipulator, including instructions to align observation latencies between the at least one sensor, an end-effector pose, and a gripper width according to measured latencies;
• generate actions for the robotic manipulator to perform a task according to the sensor data and a visuomotor policy;
• and
• control the robotic manipulator, including compensating for execution latency of the robotic manipulator in performing the actions.

A method, comprising:

• collecting sensor data about a robotic manipulator within an environment;
• pre-processing the sensor data to compensate for an observation latency of at least one sensor associated with the robotic manipulator, including aligning observation latencies between the at least one sensor, an end-effector pose, and a gripper width according to measured latencies;
• generating actions for the robotic manipulator to perform a task according to the sensor data and a visuomotor policy;
• and
• controlling the robotic manipulator, including compensating for execution latency of the robotic manipulator in performing the actions.