Robotic assembly instruction generation from a video

A device, comprising:

• one or more memories;
• and
• one or more processors, communicatively coupled to the one or more memories, configured to:
• receive a video that encodes a plurality of frames associated with assembly of an object from a plurality of sub-objects;
• determine spatio-temporal features based on the plurality of frames;
• identify a plurality of actions represented in the video based on the spatio-temporal features;
• map the plurality of actions to the plurality of sub-objects to generate an assembly plan based on the video;
• combine output from a point cloud model and output from a color embedding model to generate a plurality of sets of coordinates corresponding to the plurality of sub-objects, wherein each set of coordinates corresponds to a respective action of the plurality of actions;
• perform object segmentation to estimate a plurality of grip points and a plurality of widths corresponding to the plurality of sub-objects;
• and
• generate instructions, for one or more robotic machines for each action of the plurality of actions, based on the assembly plan, the plurality of sets of coordinates, the plurality of grip points, and the plurality of widths,
• wherein the one or more processors, to combine the output from the point cloud model and the output from the color embedding model, are configured to:
• generate a pixel-wise dense fusion matrix using the output from the point cloud model and the output from the color embedding model;
• and
• generate global features based on pooling the output from the point cloud model and the output from the color embedding model,
• wherein one of the plurality of sets of coordinates corresponding to one of the plurality of sub-objects is calculated based on the pixel-wise dense fusion matrix and the global features.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• receive a video that encodes a plurality of frames associated with assembly of an object from a plurality of sub-objects;
• determine spatio-temporal features based on the plurality of frames;
• identify a plurality of actions represented in the video based on the spatio-temporal features;
• map the plurality of actions to the plurality of sub-objects to generate an assembly plan based on the video;
• combine output from a point cloud model and output from a color embedding model to calculate a plurality of sets of coordinates corresponding to the plurality of sub-objects, wherein each set of coordinates corresponds to a respective action of the plurality of actions;
• perform object segmentation to estimate a plurality of grip points and a plurality of widths corresponding to the plurality of sub-objects;
• and
• generate instructions, for one or more robotic machines for each action of the plurality of actions, based on the assembly plan, the plurality of sets of coordinates, the plurality of grip points, and the plurality of widths,
• wherein the one or more instructions, that cause the device to combine the output from the point cloud model and the output from the color embedding model, cause the device to:
• generate a pixel-wise dense fusion matrix using the output from the point cloud model and the output from the color embedding model;
• and
• generate global features based on pooling the output from the point cloud model and the output from the color embedding model,
• wherein one of the plurality of sets of coordinates corresponding to one of the plurality of sub-objects is calculated based on the pixel-wise dense fusion matrix and the global features.