Robot tidying into non-standard categories

A computing apparatus comprising:

• a processor;
• and
• a memory storing instructions that, when executed by the processor, configure the apparatus to:
• receive, at a mobile device camera, a live video feed comprising a plurality of images capturing an environment to be tidied;
• run a panoptic segmentation model to assign a semantic label, an instance identifier, and a movability attribute to each pixel in each image of the live video feed, thereby producing a segmented image for each scene,
• wherein the instance identifier uniquely identifies the pixel for ease of reference, and
• wherein the movability attribute identifies the pixel as representing a static object, a movable object, or a tidyable object;
• detect static objects, movable objects, and tidyable objects from each segmented image based on the movability attribute, wherein reidentification fingerprints are captured for each object identified in the segmented image;
• remove the movable objects and the tidyable objects from each scene to create a static scene;
• generate keypoints for the static scene;
• generate a local static point cloud including a grid of points from the pixels representing the static objects and the keypoints from the static scene;
• compare the reidentification fingerprints for each static object in the static scene against reidentification fingerprints for known static objects to detect visual matches to the known static objects, wherein the reidentification fingerprints for the known static objects are stored in a global database in communication with the mobile device;
• determine matches between the local static point cloud and a global point cloud using matching static objects and matching the keypoints from the static scene;
• determine a current pose of the mobile device camera relative to a global map, wherein the global map is a previously saved map of the environment to be tidied;
• merge the local static point cloud into the global point cloud and removing duplicate points to update the current pose of the mobile device camera on the global map;
• save a location record for each static object to the global database, wherein the location record includes a timestamp and a location of each static object on the global map,
• on condition the location record for a static object is inconsistent with past location records stored on the global database for the static object, indicate that the static object has been moving:
• generate an inconsistent static object location alert;
• provide the inconsistent static object location alert to a robotic control system of a tidying robot as feedback to the robotic control system to instruct the tidying robot to perform at least one robot operation;
• reclassify the static object as a reclassified movable object by updating the movability attribute in the global database;
• and
• update the global map to reflect the reclassified movable object;
• prioritize operational task rules based on at least one of the movability attributes and the updated movability attributes;
• and
• instruct the robotic control system of the tidying robot to perform the at least one robot operation utilizing the operational task rules and a robot instruction database.

A method comprising:

• receiving, at a mobile device camera, a live video feed capturing an environment to be tidied, wherein the mobile device is one of a mobile computing device operated by a user and a tidying robot;
• processing the live video feed, the live video feed comprising images of scenes, to display an augmented reality view to the user of a global map of the environment to be tidied;
• running a panoptic segmentation model to assign a semantic label, an instance identifier, and a movability attribute to each pixel in each image, thereby producing a segmented image for each scene,
• wherein the instance identifier uniquely identifies the pixel for ease of reference, and
• wherein the movability attribute identifies the pixel as representing a static object, a movable object, or a tidyable object;
• separating, from the segmented image for each scene, static objects from movable objects and tidyable objects;
• generating reidentification fingerprints, in each scene, for each static object, each movable object, and each tidyable object;
• placing the reidentification fingerprints into a global database including known static objects, known movable objects, and known tidyable objects;
• generating keypoints for a static scene with each movable object and each tidyable object removed;
• determining a basic room structure using segmentation, wherein the basic room structure includes at least one of a floor, a wall, and a ceiling;
• determining an initial pose of the mobile device camera relative to a floor plane;
• generating a local static point cloud including a grid of points from the pixels representing the static objects and the keypoints from the static scene;
• comparing each static object in the static scene against the global database to find a visual match to the known static objects using the reidentification fingerprints;
• determining matches between the local static point cloud and a global point cloud using matching static objects and matching the keypoints from the static scene;
• determining a current pose of the mobile device camera relative to the global map, wherein the global map is a previously saved map of the environment to be tidied;
• merging the local static point cloud into the global point cloud and removing duplicate points to update the current pose of the mobile device camera on the global map;
• saving, to the global database, a location of each static object on the global map and a timestamp;
• updating the global database with an expected location of each static object on the global map based on past location records,
• on condition the past location records are inconsistent for a static object indicating that the static object has been moving:
• generating an inconsistent static object location alert;
• providing the inconsistent static object location alert to a robotic control system of the tidying robot as feedback to the robotic control system to instruct the tidying robot to perform at least one robot operation;
• reclassifying the static object as a reclassified movable object by updating the movability attribute in the global database;
• and
• updating the global map to reflect the reclassified movable object;
• generating a local movable point cloud using a center coordinate of each movable object;
• using the current pose of the mobile device camera on the global map to convert the local movable point cloud to a global coordinate frame;
• comparing each movable object in the scene against the global database to find a visual match to the known movable objects using the reidentification fingerprints;
• and
• saving, to the global database, a location of each movable object on the global map and the timestamp;
• generating a local tidyable point cloud using a center coordinate of each tidyable object;
• using the current pose of the mobile device camera on the global map to convert the local tidyable point cloud to the global coordinate frame;
• comparing each tidyable object in the scene against the global database to find a visual match to the known tidyable objects using the reidentification fingerprints;
• and
• saving, to the global database, a location of each tidyable object on the global map and the timestamp;
• determining a bounded area on the global map by at least one of:
• receiving a bounded area selection signal from the user;
• and
• detecting areas bounded by static objects;
• determining a label for the bounded area to create a named bounded area by at least one of:
• receiving a label selection signal from the user;
• and
• generating the label based on at least one of the static objects, the movable objects, and the tidyable objects identified in the bounded area;
• defining at least one operational task rule that is an area-based rule using the named bounded area,
• wherein the area-based rule is at least one of:
• a time rule controlling performance of the at least one robot operation based on the timestamp;
• an object rule controlling the performance of the at least one robot operation based on detecting a specific tidyable object;
• and
• a category rule controlling the performance of the at least one robot operation based on detecting a tidyable object of a specific category;
• wherein the area-based rule controls the performance of the at least one robot operation when the tidying robot is located in the named bounded area;
• prioritizing operational task rules based on at least one of the movability attributes and the updated movability attributes;
• and
• instructing the tidying robot to perform the at least one robot operation utilizing the operational task rules and a robot instruction database.