System and method for imitation learning in robotics for complex task learning

A system comprising:

• a memory storing instructions;
• one or more Input/Output (I/O) interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
• collect a plurality of videos of a long-horizon task demonstration and a plurality of kinesthetic robot demonstration, wherein during the kinesthetic robot demonstration a manipulator robotic arm is manually actuated to perform a plurality of primitive actions and joint angle data of the manipulator is recorded;
• sampling a plurality of frames at a plurality of time steps from each of the plurality of videos using a uniform temporal sampling to identify a sequence of one or more sub-tasks in the plurality of videos of a task demonstration;
• label the sampled plurality of frames with an associated sub-task of the one or more sub-tasks to generate a training dataset;
• train a Task Sequencing Network (TSNet) using the generated training dataset comprising the long-horizon task demonstration to predict the sequence of one or more sub-tasks associated with the plurality of frames;
• obtain the one or more sub-tasks predicted with highest probability at each of the plurality of time steps, and removing one or more duplicate sub-tasks and extra blank tokens to predict the sequence of one or more sub-tasks using the trained TSNet;
• build a manipulation graph from the predicted sequence of one or more sub-tasks to generate a parameterized trajectory based on the manipulation graph;
• train one or more Dynamic Movement Primitive (DMP) models using the plurality of the kinesthetic robot task demonstration to build a Task Agnostic DMP Library (TADL) comprising DMPs associated with the one or more sub-tasks;
• and
• select a relevant movement for the one or more sub-tasks from the TADL to execute an intended movement comprising of one or more sub-tasks on a manipulator robotic arm.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• collecting, via an input/output interface, a plurality of videos of a long-horizon task demonstration and a plurality of kinesthetic robot demonstration, wherein during the kinesthetic robot demonstration a manipulator robotic arm is manually actuated to perform a plurality of primitive actions and joint angle data of the manipulator is recorded;
• sampling a plurality of frames at a plurality of time steps from each of the plurality of videos using a uniform temporal sampling to identify a sequence of one or more sub-tasks in the plurality of videos of a task demonstration;
• labeling the sampled plurality of frames with an associated sub-task of the one or more sub-tasks to generate a training dataset;
• training a Task Sequencing Network (TSNet) using the generated training dataset comprising the long-horizon task demonstration to predict the sequence of one or more sub-tasks associated with the plurality of frames;
• obtaining the one or more sub-tasks predicted with highest probability at each of the plurality of time steps, and removing one or more duplicate sub-tasks and extra blank tokens to predict the sequence of one or more sub-tasks using the trained TSNet;
• building a manipulation graph from the predicted sequence of one or more sub-tasks to generate a parameterized trajectory based on the manipulation graph;
• training one or more Dynamic Movement Primitive (DMP) models using the plurality of the kinesthetic robot task demonstration to build a Task Agnostic DMP Library (TADL) comprising DMPs associated with the one or more sub-tasks;
• and
• selecting a relevant movement for the one or more sub-tasks from the TADL to execute an intended movement comprising of one or more sub-tasks on a manipulator robotic arm.