Generating robotic control plans

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining, from a user device, image data comprising a plurality of images of an instruction manual, wherein the instruction manual depicts assembly actions for assembling a plurality of assembly components into a product;
• generating, for each image using a machine learning model, instruction data for the image, the instruction data representing a sequence of instructions for a robot to perform one or more assembly actions to assemble a respective plurality of assembly components represented in the image, wherein the machine learning model has been configured, through training on training examples that each include one or more illustrations of one or more assembly actions using a respective plurality of assembly components and ground-truth instruction data representing a sequence of instructions for a robot to perform the one or more assembly actions using the plurality of assembly components;
• processing the instruction data to generate a sequence of stages, each stage having a plurality of subtasks to be executed by one or more robots for assembling the plurality of assembly components identified from the depictions of assembly actions for assembling the assembly components into the product in the one or more images of the image data;
• and
• causing a robotic control system to execute the plurality of subtasks using the one or more robots.

One or more non-transitory storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining, from a user device, image data comprising a plurality of images of an instruction manual, wherein the instruction manual depicts assembly actions for assembling a plurality of assembly components into a product;
• generating, for each image using a machine learning model, instruction data for the image, the instruction data representing a sequence of instructions for a robot to perform one or more assembly actions to assemble a respective plurality of assembly components represented in the image, wherein the machine learning model has been configured, through training on training examples that each include one or more illustrations of one or more assembly actions using a respective plurality of assembly components and ground-truth instruction data representing a sequence of instructions for a robot to perform the one or more assembly actions using the plurality of assembly components;
• processing the instruction data to generate a sequence of stages, each stage having a plurality of subtasks to be executed by one or more robots for assembling the plurality of assembly components identified from the depictions of assembly actions for assembling the assembly components into the product in the one or more images of the image data;
• and
• causing a robotic control system to execute the plurality of subtasks using the one or more robots.