Method and system for generating robotic instructions

A system for generating robotic instructions from contextual data, the system comprising:

• at least one memory storing instructions;
• and
• at least one processor communicatively coupled with the at least one memory and configured to execute the instructions to perform operations comprising:
• receiving video data demonstrating one or more tasks;
• receiving text data related to the one or more tasks;
• encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer;
• receiving image data providing environmental data for at least one robotic task;
• encoding vision data corresponding to the image data;
• and
• generating, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions.

A non-transitory computer-readable media (CRM) storing instructions thereon, which, when executed by at least one processor of a computing device, cause the computing device to generate robotic instructions from contextual data by performing operations comprising:

• receiving video data demonstrating one or more tasks;
• receiving text data related to the one or more tasks;
• encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer;
• receiving image data providing environmental data for at least one robotic task;
• encoding vision data corresponding to the image data;
• and
• generating, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions.