Dynamically refining markers in an autonomous world model

A system for performing autonomous operations in an operating environment comprising a plurality of objects, the system comprising:

• an autonomous device comprising:
• a first non-transitory computer readable medium storing detailed representations of the objects in the operating environment, wherein each detailed representation of an object comprises one or more markers providing knowledge information related to the object and pertaining to how to manipulate the object by the autonomous device,
• a second non-transitory computer readable medium storing a world model that comprises sparse representations of the objects in the operating environment, wherein the sparse representations of the objects include less information than the detailed representations of the objects stored in the first non-transitory computer readable medium,
• a processor configured to use the world model to execute tasks requiring use of at least one of the objects, and
• wherein, when executing a new task requiring use of a particular object in the operating environment, the autonomous device updates the world model to include the detailed representation of the particular object including the one or more markers.

A method for performing autonomous operations in an operating environment comprising a plurality of objects, the method comprising:

• storing, in a first memory at a server computer, detailed representations of the objects in the operating environment, wherein the detailed representation of an object comprises one or more markers providing knowledge information related to the object and pertaining to how to manipulate the object by an autonomous device;
• storing, in a second memory at the autonomous device, a world model comprising sparse representations of the objects in the operating environment, wherein the sparse representations of the objects include less information than the detailed representations of the objects stored in the first memory;
• receiving, by the autonomous device, a new task requiring use of a particular object in the operating environment;
• retrieving, by the autonomous device, a detailed representation of the particular object from the server computer in response to receiving the new task;
• updating, by the autonomous device, the world model to include the detailed representation of the particular object including the one or more markers;
• and
• utilizing, by the autonomous device, the updated world model to perform the new task.