Clutter tidying robot utilizing floor segmentation for mapping and navigation system

An apparatus comprising:

• a robot;
• a processor;
• and
• a memory storing instructions that, when executed by the processor, configure the apparatus to:
• receive, at a perception module configured in operational logic of a robotic control system, image data from the robot's sensors,
• wherein the perception module comprises a detection module, a scene segmentation module, and a mapping module,
• wherein the robot's sensors include at least one of cameras, lidar sensor, inertial measurement unit (IMU) sensors, wheel encoder, and other sensors, and
• wherein the operational logic of the robotic control system comprises executable instructions stored by a storage device of the robotic control system and executed by a processor of the robotic control system;
• detect, by the detection module, objects from the image data, as two-dimensional (2D) bounding boxes with object classes;
• generate, by the detection module:
• predicted three-dimensional (3D) object locations, using the 2D bounding boxes and a ground plane;
• and
• two-dimensional-three-dimensional (2D-3D) bounding boxes with class labels, the 2D-3D bounding boxes based on the 2D bounding boxes and the predicted 3D object locations;
• generate, by the scene segmentation module:
• a multi-class segmentation map, using a segmentation model to segregate a floor boundary and other relevant regions in the image data;
• an edge map including the floor boundary and other relevant boundaries, with semantic information;
• and
• semantic boundary masks, from the multi-class segmentation map, wherein the semantic boundary masks identify relevant boundaries and their semantic information;
• generate, by the mapping module, using the 2D-3D bounding boxes and the semantic boundary masks:
• a scene layout map, wherein the scene layout map includes global elements relevant for global mapping;
• and
• a local occupancy map, wherein the local occupancy map includes local elements useful for local path planning and local obstacle avoidance;
• receive, at a navigation module configured in the operational logic of the robotic control system, the scene layout map, and the local occupancy map, wherein the navigation module includes:
• a simultaneous localization and mapping module (SLAM);
• a global mapper module;
• and
• a fusion and inflation module;
• generate, by the SLAM, using lidar data, (IMU) data, and wheel encoding data:
• a SLAM global map, which comprises a 2D occupancy grid representation of an environment with obstacle information at lidar height and real-time location information of the robot;
• and
• a robot pose;
• generate, by the global mapper module, using the SLAM global map, the lidar data, and the scene layout map:
• a navigation global map, which represents an improved 2D occupancy grid representation of the environment over the SLAM global map;
• generate, by the fusion and inflation module using the lidar data, the navigation global map, and the local occupancy map:
• a fused local occupancy map, with the lidar data and information from the navigation global map and the local occupancy map, making the fused local occupancy map useful for obstacle avoidance;
• an inflated global map, which includes buffer regions for the robot around obstacles;
• and
• an inflated local map based on the fused local occupancy map, which includes the buffer regions for the robot around the obstacles;
• receive, at a robot brain, the inflated global map, the robot pose, and the inflated local map;
• generate, by the robot brain configured in the operational logic of the robotic control system, robot action commands based on at least one of the inflated global map, the robot pose, and the inflated local map;
• receive, at an action module configured in the operational logic of the robotic control system, the robot action commands;
• and
• control robot actuators in response to the robot action commands.