Methods and systems for generating instructions for a robotic system to carry out a task

A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a computing system to perform functions comprising:

• receiving a query from a first robotic device having a plurality of sensors, wherein the query comprises sensor data obtained from the plurality of sensors, a request for instructions to carry out a task, and information associated with a configuration of the first robotic device;
• identifying stored data comprising a procedure previously used by a second robotic device to carry out the task or a similar task and information associated with a configuration of the second robotic device;
• generating instructions for the first robotic device to carry out the task based at least in part on the sensor data, the procedure used by the second robotic device to carry out the task or the similar task, the information associated with the configuration of the first robotic device, and the information associated with the configuration of the second robotic device;
• execute a simulation based on the generated instructions;
• obtaining results of the simulation and responsively determining a confidence measure, based on the results, related to a measure that the generated instructions allow for successful performance of the task;
• and
• generating updated instructions to carry out the task, by the first robotic device, based on the confidence measure being below a threshold.

A method comprising:

• receiving a query from a first robotic system having one or more sensors, wherein the query comprises sensor data obtained from the one or more sensors, a request for instructions to carry out an environment interaction, and information associated with a configuration of the first robotic system;
• identifying stored data comprising a procedure previously used by a second robotic system to carry out the environment interaction and information associated with a configuration of the second robotic system;
• generating instructions for the first robotic system to carry out the environment interaction based at least in part on the sensor data, the procedure used by the second robotic system to carry out the environment interaction, the information associated with the configuration of the first robotic system, and the information associated with the configuration of the second robotic system;
• conducting an iterative procedure until a confidence measure is above a threshold, wherein the confidence measure is related to a measure that the generated instructions allow for successful performance of the environment interaction, and wherein iterations of the iterative procedure comprise:
• executing a simulation based on the generated instructions;
• obtaining results of the simulation and responsively determining the confidence measure based on the results;
• and
• updating the generated instructions to carry out the environment interaction, by the first robotic system, based on the confidence measure being below the threshold.