Techniques for vision-based robot control

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• generating, based on scene data, a plurality of scenes;
• generating, based on the plurality of scenes, one or more goal specifications;
• determining, based on the one or more goal specifications and a robot model, one or more robot plans;
• generating, based on the one or more robot plans and the plurality of scenes, simulated sensor data;
• and
• performing one or more training operations to generate a trained vision-based robot control model based on the one or more goal specifications, the one or more robot plans, and the simulated sensor data.

A system comprising:

• one or more memories storing instructions, and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
• generate, based on scene data, a plurality of scenes,
• generate, based on the plurality of scenes, one or more goal specifications,
• determine, based on the one or more goal specifications and a robot model, one or more robot plans,
• generate, based on the one or more robot plans and the plurality of scenes, simulated sensor data, and
• perform one or more training operations to generate a trained vision-based robot control model based on the one or more goal specifications, the one or more robot plans, and the simulated sensor data.