Dynamic coordination of multiple robotic manipulator arms

A non-transitory computer-readable medium on which is stored instructions for controlling a robotic device, which, when executed, cause a processor to perform operations comprising:

• acquiring, from a perspective of a robotic device, image data of a real-world environment associated with a plurality of tasks to be performed in coordination by at least a first robotic manipulator arm and a second robotic manipulator arm of the robotic device;
• identifying one or more objects in the image data;
• generating one or more parametric representations of the one or more objects in the image data;
• generating, using the one or more parametric representations, a parametric model of the real-world environment associated with the plurality of tasks, wherein the parametric model includes a three-dimensional volumetric grid composed of voxels, and wherein each voxel is assigned with a low-dimensional representation of a state of a portion of the real-world environment and semantic information associated with the one or more objects;
• based on the parametric model, determining a planned trajectory of the first robotic manipulator arm over a predetermined number of time steps, including identifying whether an obstacle will intersect with the planned trajectory of the first robotic manipulator arm within the predetermined number of time steps;
• controlling the first robotic manipulator arm to execute the planned trajectory according to a first task of the plurality of tasks including manipulating a first object of the one or more objects to reveal a second object in the real-world environment;
• dynamically updating the parametric model of the real-world environment associated with the plurality of tasks;
• and
• controlling, based on the updated parametric model, the second robotic manipulator arm to perform a second task of the plurality of tasks on the second object, wherein the second task is performed in coordination with the first task.

An apparatus comprising:

• a plurality of robotic manipulator arms;
• one or more image sensors;
• a processor;
• and
• a memory storing instructions, which when executed by the processor, result in operations comprising:
• acquiring, from the one or more image sensors, image data of a real-world environment associated with a plurality of tasks to be performed in coordination by at least a first robotic manipulator arm and a second robotic manipulator arm of the plurality of robotic manipulator arms;
• identifying one or more objects in the image data;
• generating one or more parametric representations of the one or more objects in the image data;
• generating, using the one or more parametric representations, a parametric model of the real-world environment associated with the plurality of tasks, wherein the parametric model includes a three-dimensional volumetric grid composed of voxels, and wherein each voxel is assigned with a low-dimensional representation of a state of a portion of the real-world environment and semantic information associated with the one or more objects;
• based on the parametric model, determining a planned trajectory of the first robotic manipulator arm over a predetermined number of time steps, including identifying whether an obstacle will intersect with the planned trajectory of the first robotic manipulator arm within the predetermined number of time steps;
• controlling the first robotic manipulator arm to execute the planned trajectory according to a first task of the plurality of tasks including manipulating a first object of the one or more objects to reveal a second object in the real-world environment;
• dynamically updating the parametric model of the real-world environment associated with the plurality of tasks;
• and
• controlling, based on the updated parametric model, the second robotic manipulator arm to perform a second task of the plurality of tasks on the second object, wherein the second task is performed in coordination with the first task.