Training with high fidelity simulations and high speed low fidelity simulations

A method implemented using one or more processors, comprising:

• during a first phase of training a robot control policy for controlling a robot:
• training the robot control policy based on a first set of training data, wherein the first set of training data comprises, in a first ratio, (i) training data generated based on simulated operation of the robot in a first fidelity simulation, and (ii) training data generated based on simulated operation of the robot in a second fidelity simulation, wherein the second fidelity is greater than the first fidelity;
• and
• determining that one or more criteria for commencing a second phase of training the robot control policy are satisfied, wherein the one or more criteria for commencing the second phase of training the robot control policy comprise a determination that learning of the robot control policy from training using the first set of training data has slowed by at least a first threshold;
• during the second phase of training the robot control policy:
• training the robot control policy based on a second set of training data, wherein the second set of training data comprises, in a second ratio, (i) training data generated based on simulated operation of the robot in the first fidelity simulation, and (ii) training data generated based on simulated operation of the robot in the second fidelity simulation, wherein the proportion of training data generated based on simulated operation of the robot in the first fidelity simulation relative to training data generated based on simulated operation of the robot in the second fidelity simulation in the second ratio is lower than in the first ratio;
• wherein the first fidelity simulation is rendered with a lower resolution than the second fidelity simulation, and generating training data based on the first fidelity simulation comprises:
• capturing sensor data at the lower resolution of the first fidelity simulation;
• and
• upscaling the sensor data to a resolution expected by the robot control policy.

A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:

• during a first phase of training a robot control policy for controlling a robot:
• train the robot control policy based on a first set of training data, wherein the first set of training data comprises, in a first ratio, (i) training data generated based on simulated operation of the robot in a first fidelity simulation, and (ii) training data generated based on simulated operation of the robot in a second fidelity simulation, wherein the second fidelity is greater than the first fidelity;
• determine that one or more criteria for commencing a second phase of training the robot control policy are satisfied, wherein the one or more criteria for commencing the second phase of training the robot control policy comprise a determination that learning of the robot control policy from training using the first set of training data has slowed by at least a first threshold;
• and
• during the second phase of training the robot control policy:
• train the robot control policy based on a second set of training data, wherein the second set of training data comprises, in a second ratio, (i) training data generated based on simulated operation of the robot in the first fidelity simulation, and (ii) training data generated based on simulated operation of the robot in the second fidelity simulation, wherein the proportion of training data generated based on simulated operation of the robot in the first fidelity simulation relative to training data generated based on simulated operation of the robot in the second fidelity simulation in the second ratio is lower than in the first ratio;
• and
• subsequent to the second phase of training the robot control policy, train the robot control policy based on a third set of training data that is entirely comprised of training data generated based on simulated operation of the robot in the second fidelity simulation.