Devices, systems, and methods for enhanced robotic manipulation and control

A system for enhanced robotic manipulation and control relative to an object of interest within an environment, the system comprising:

• a mobile robot navigating the environment, the mobile robot comprising:
• a sensor generating sensor data indicative of a physical attribute of the object of interest;
• and
• a robotic arm comprising an actuator, wherein the actuator causes the robotic arm to move relative to the object of interest;
• and
• a computer system communicably coupled to the mobile robot, wherein the computer system comprises a processor and a memory storing an algorithm that, when executed by the processor, causes the computer system to:
• receive sensor data generated by the sensor;
• detect the object of interest based on the sensor data;
• determine whether the detected object of interest meets or exceeds a first threshold based on the sensor data;
• generate, upon determining that the object of interest meets or exceeds the first threshold, a label associated with a keypoint of the object of interest based on the sensor data;
• determine an orientation and scale of the object of interest in the environment relative to the robotic arm based on the generated label;
• generate a motion plan for the robotic arm based on the determined orientation and scale of the object of interest;
• and
• control the actuator of the robotic arm to cause the robotic arm to perform the generated motion plan.