Interpreting discrete tasks from complex instructions for robotic systems and applications

A system, comprising:

• a language model to convert a spoken instruction into a textual representation of a compound task;
• an image model to determine a set of visual features corresponding to a current state represented in an image of an environment in which the compound task is to be performed;
• a natural language understanding module to determine a set of correlations between both the textual representation of the spoken request and the set of visual features corresponding to a current state, the set of correlations being determined using natural language understanding on the textual representation of the spoken request and on the set of visual features;
• a task planner to determine a sequence of discrete tasks to be performed for the compound task, based on both the set of correlations and a set of performable actions;
• and
• an execution module to cause instructions for the sequence of discrete tasks to be performed.

A non-transitory computer-readable storage medium including instructions that, if executed by one or more processors, cause the one or more processors to:

• analyze audio data to generate a textual representation of a spoken request, the spoken request corresponding to a performance of a compound task represented using the audio data;
• analyze image data to obtain a set of current conditions for a current state of an environment;
• obtain a set of expected conditions of the environment by processing the textual representation of the spoken request together with one or more features extracted from the image data representing the current state, wherein the processing includes using natural language understanding on the textual representation of the spoken request and on the one or more features extracted from the image data;
• determine, based at least in part upon a set of performable actions, a sequence of discrete tasks to be performed to transition from the set of current conditions to the set of expected conditions of the environment;
• and
• cause instructions for the sequence of discrete tasks to be executed to perform the compound task.