Simulation driven robotic control of real robot(s)

A method implemented by one or more processors, the method comprising:

• configuring a simulated environment, of a robotic simulator that simulates the simulated environment, based on real environment state data that is based on one or more sensor-based observations of a real environment;
• determining, using the robotic simulator, a candidate sequence of robotic actions for use in controlling a real robot, in the real environment, in performing at least part of a robotic task, wherein determining the candidate sequence of robotic actions comprises:
• processing simulated state data that simulates real state data that is fully incapable of being generated utilizing any sensors in the real environment, wherein the simulated state data is generated from a simulated viewpoint that corresponds to a real viewpoint, in the real environment, and wherein none of the sensors in the real environment are capable of, without human intervention, capturing real vision data from the real viewpoint, in the real environment, that corresponds to the simulated viewpoint from which the simulated state data is generated;
• and
• causing the real robot to implement the candidate sequence of robotic actions.

A system, comprising:

• memory storing instructions;
• one or more processors executing the instructions to:
• configure a simulated environment, of a robotic simulator that simulates the simulated environment and a simulated robot, based on real environment state data that is based on one or more sensor-based observations of a real environment;
• determine, using the robotic simulator, a first candidate sequence of robotic actions and a second candidate sequence of robotic actions, for use in controlling a real robot, in the real environment, in performing at least part of a robotic task;
• apply, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the first candidate sequence of robotic actions to generate first simulated state data;
• apply, to the simulated robot of the robotic simulator and with the simulated environment after the configuring, the second candidate sequence of robotic actions to generate second simulated state data;
• determine, based on the first simulated state data and the second simulated state data, to utilize the first candidate sequence of robotic actions, in lieu of the second candidate sequence of robotic actions, for use in controlling the real robot in performing at least part of the robotic task;
• determine, prior to causing the real robot to implement the first candidate sequence of robotic actions and based on comparing a portion of the simulated environment to a corresponding portion of updated real environment state data, that a degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfies a threshold;
• and
• in response to determining to utilize the first candidate sequence of robotic actions and based on the degree of similarity between the portion of the simulated environment and the corresponding portion of the updated real environment state data satisfying the threshold:
• cause the real robot to implement the first candidate sequence of robotic actions.