Method and system for generating a sequence of actions for controlling a robot

A system for applying a neural network including an action sequence decoder configured for generating a sequence of actions for a robot to perform a task, the neural network being applied to generate the sequence of actions based on a plurality of recordings demonstrating one or multiple humans performing one or multiple tasks, the system comprising

• a processor;
• and
• a memory having instructions stored thereon that cause the processor of the system to:
• collect a recording of the plurality of recordings and a sequence of captions describing scenes in the recording;
• extract feature data from the recording, the feature data includes features of one or a combination of a video signal, an audio signal, and text transcription capturing a performance of the task;
• encode the extracted feature data with one or a combination of a video encoder, an audio encoder, and a text encoder to produce a sequence of encoded features;
• and
• apply the action sequence decoder to produce the sequence of actions for the robot based on the sequence of encoded features having a semantic meaning corresponding to a semantic meaning of the sequence of captions describing scenes in the recording,
• wherein the sequence of actions is aligned with the sequence of captions to have a one-to-one correspondence between the actions and the captions.

A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a system, causes a processor of the system to execute operations for applying a neural network including an action sequence decoder configured for generating a sequence of actions for a robot to perform a task, the neural network being applied to generate the sequence of actions based on a plurality of recordings demonstrating one or multiple humans performing one or multiple tasks, the operations comprising:

• collecting a recording of the plurality of recordings and a sequence of captions describing scenes in the recording;
• extracting feature data from the recording, the feature data includes features of one or a combination of a video signal, an audio signal, and text transcription capturing a performance of the task;
• encoding the extracted feature data with one or a combination of a video encoder, an audio encoder, and a text encoder to produce a sequence of encoded features;
• and
• applying the action sequence decoder to produce the sequence of actions for the robot based on the sequence of encoded features having a semantic meaning corresponding to a semantic meaning of the sequence of captions describing scenes in the recording,
• wherein the sequence of actions is aligned with the sequence of captions to have a one-to-one correspondence between the actions and the captions.