System and method for generating unified goal representations for cross task generalization in robot navigation

A system for a machine-learning network comprising:

• one or more processors configured to:
• receive, by a device, a command from a user related to a subject;
• access a representation space associated with the command, where similar subjects and commands in the representation space are clustered together;
• receive a first dataset related to the command, a second dataset related to the subject, and a third dataset which includes subjects related to the command;
• update the representation space based on at least one of the first dataset, the second dataset, and the third dataset;
• generate, by a goal description machine learning model, a goal representation based on the representation space;
• receive, from a plurality of sensors, a sensor data of a current environment;
• generate a first series of steps and a second series of steps based on the goal representation and the current environment;
• annotate, by a progress description machine learning model, the sensor data based on performance of the first series of steps to generate an annotated senor data;
• and
• update, by a policy machine learning model, the second series of steps based on the annotated sensor data.

A machine-learning network for a machine-learning network comprising:

• one or more processors configured to:
• receive, by a device, a command from a user related to a subject;
• access a representation space associated with the command, where similar subjects and commands in the representation space are clustered together;
• receive a first dataset related to the command, a second dataset related to the subject, and a third dataset which includes subjects related to the command;
• update the representation space based on at least one of the first dataset, the second dataset, and the third dataset;
• generate, by a goal description machine learning model, a goal representation based on the representation space;
• receive, from a plurality of sensors, a sensor data of a current environment;
• generate a first series of steps and a second series of steps based on the goal representation and the current environment;
• annotate, by a progress description machine learning model, the sensor data based on performance of the first series of steps to generate an annotated senor data;
• and
• update, by a policy machine learning model, the second series of steps based on the annotated sensor data.