Systems and methods for operating robots using object-oriented Partially Observable Markov Decision Processes

A system comprising:

• a mobile robot having at least one processor configured to execute computer readable instructions stored in at least one non-transitory computer readable storage medium to perform a multi-object search task;
• a representation of a multi-object search task in an Object-Oriented Partially Observable Markov Decision Process model having at least one belief pertaining to a state and at least one observation space within an environment of the robot, wherein the state is represented in terms of classes and objects and each object has at least one attribute and a semantic label;
• a command module communicatively coupled to the at least one processor and configured to receive from a user, a language command identifying at least one target object and at least one location corresponding to the target object;
• an update module communicatively coupled to the at least one processor and configured to update the at least one belief, associated with the at least one target object, based on the language command;
• a drive module coupled to the mobile robot and configured to enable driving the mobile robot to the at least one observation space identified in the updated belief,
• at least one sensor coupled to the mobile robot, the at least one sensor configured to search and provide a sensor signal to the at least one processor upon finding the at least one target object;
• and
• a notification module communicatively coupled to the processor and configured to notify the user upon finding the at least one target object.

A method of operating a mobile robot to conduct a task, the mobile robot having at least one processor executing computer readable instructions stored in at least one non-transitory computer readable storage medium to perform operations comprising:

• representing a task in an Object-Oriented Partially Observable Markov Decision Process model having at least one belief pertaining to a state and at least one observation space within an environment of the robot, wherein the state is represented in terms of classes and objects and each object has at least one attribute and a semantic label;
• receiving, at the mobile robot from a user, a language command identifying at least one target object and at least one location corresponding to the target object;
• updating the at least one belief, associated with the at least one target object, based on the language command;
• generating at least one action using the language command and the updated belief, and executing the at least one action.

A computer program, embodied on at least one non-transitory computer readable medium, the computer program, when executed by a robot having at least one processor, causes the robot to perform operations comprising:

• storing at least one belief pertaining to a state and at least one observation space within an environment of the robot, wherein the state is represented in terms of classes and objects and each object has at least one attribute and a semantic label;
• receiving a language command from a user, the language command identifying at least one target object and at least one location corresponding to the target object;
• updating the at least one belief, associated with the at least one target object, based on the language command;
• driving the mobile robot to the at least one observation space identified in the updated belief;
• searching, using at least one sensor on the mobile robot while traversing the at least one observation space identified in the updated belief, for the at least one target object;
• and
• notifying the user upon finding the at least one target object.