Robotic task planning for complex task instructions in natural language

A system for robotic task planning for a complex instruction in natural language to execute an intended task, the system comprising:

• one or more data storage devices operatively coupled to one or more hardware processors and configured to store instructions configured for execution via the one or more hardware processors to:
• receive a set of tokens I={w1, w2, . . . wn}representative of tasks in an original task sequence w1:
• n, present in the complex instruction and a set of task type labels T={t1, t2, . . . tn} in a task type label sequence t1:
• m, wherein each of the tasks is associated with (i) a corresponding task type label in the set of task type labels T and (ii) one or more arguments that are grounded to objects, locations and states of the objects in an environment of a robot performing the tasks;
• predict, a set of task dependency labels D={d1, d2, . . . dm} in a task dependency label sequence d1:
• m corresponding to the task type label sequence t1:
• m using a pre-trained Conditional Random Field (CRF) model;
• and
• generate a Conditional Planning Tree (CPT) with a resolved task sequence by modifying the original task sequence w1:
• n based on (i) the task type label, (ii) an argument from the one or more arguments and (iii) the predicted task dependency label associated with each of the tasks present in the complex instruction to execute the intended task.

A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

• receive a set of tokens I={w1, w2, . . . wn} representative of tasks in an original task sequence w1:
• n, present in the complex instruction and a set of task type labels T={t1, t2, . . . tn} in a task type label sequence t1:
• m, wherein each of the tasks is associated with (i) a corresponding task type label in the set of task type labels T and (ii) one or more arguments that are grounded to objects, locations and states of the objects in an environment of a robot performing the tasks;
• predict, a set of task dependency labels D={d1, d2, . . . dm} in a task dependency label sequence d1:
• m corresponding to the task type label sequence t1:
• m using a pre-trained Conditional Random Field (CRF) model;
• and
• generate a Conditional Planning Tree (CPT) with a resolved task sequence by modifying the original task sequence w1:
• n based on (i) the task type label, (ii) an argument from the one or more arguments and (iii) the predicted task dependency label associated with each of the tasks present in the complex instruction to execute the intended task.