Systems and Methods for Automated Response to Natural Language Instructions

A method for enabling a machine to act upon natural language instructions, comprising:

• obtaining a plurality of instruction and observation pairs;
• generating language embeddings for each instruction in the plurality of instruction and observation pairs using a language encoder;
• generating observation embeddings for each observation in the plurality of instruction and observation pairs;
• predicting a set of skill codes for each given pair in the plurality of instruction and observation pairs based on a given language embedding and a given observation embedding generated from the given pair using a skill predictor;
• predicting an action to correctly resolve the instruction of the given pair using a policy based on the set of skill codes;
• and
• controlling a device to perform the predicted action.

A system for enabling a machine to act upon natural language instructions, comprising:

• a processor;
• a controllable device;
• and
• a memory, comprising a natural language processing application that configures the processor to:
• obtain a plurality of instruction and observation pairs;
• generate language embeddings for each instruction in the plurality of instruction and observation pairs using a language encoder;
• generate observation embeddings for each observation in the plurality of instruction and observation pairs;
• predict a set of skill codes for each given pair in the plurality of instruction and observation pairs based on a given language embedding and a given observation embedding generated from the given pair using a skill predictor;
• predict an action to correctly resolve the instruction of the given pair using a policy based on the set of skill codes;
• and
• control the controllable device to perform the predicted action.