Methods and systems for remote perception assistance to facilitate robotic object manipulation

A method comprising:

• identifying, from a model of objects in an environment of a robotic manipulator, a virtual boundary line separating two adjacent identified virtual objects, wherein the robotic manipulator is configured to perform a task involving a manipulation of at least one object in the environment represented by the identified virtual objects;
• based on the identifying, sending, to a remote assistor device, a request for remote assistance with distinguishing the at least one object in the model, wherein the request includes information indicative of the virtual boundary line separating the two adjacent identified virtual objects, wherein the remote assistor device is configured to display a graphical user interface (GUI) that overlays the virtual boundary line on the model, and wherein the GUI allows for (i) a confirmation of the virtual boundary line and (ii) a modification to the virtual boundary line;
• receiving, from the remote assistor device, a response to the request, wherein the response comprises one of (i) the confirmation of the virtual boundary line and (ii) the modification to the virtual boundary line;
• and
• causing the robotic manipulator to perform the task based on the response.

A remote assistor computing device for providing remote assistance to a control system of a robotic manipulator, the remote assistor computing device comprising:

• a display;
• at least one processor;
• and
• data storage comprising instructions executable by the at least one processor to cause the remote assistor computing device to perform operations comprising:
• receiving, from the control system, a request for remote assistance with distinguishing at least one object in a model of objects in an environment of the robotic manipulator, wherein the request includes information indicative of a virtual boundary line separating two adjacent identified virtual objects, and wherein the robotic manipulator is configured to perform a task involving a manipulation of the at least one object in the environment represented by the identified virtual objects;
• in response to receiving the request, providing, on the display, a graphical user interface (GUI) representative of the model and the virtual boundary line, wherein the virtual boundary line is overlaid on at least a portion of the model, and wherein the GUI allows for (i) a confirmation of the virtual boundary line and (ii) a modification to the virtual boundary line;
• receiving, via the GUI, input data indicating one of (i) the confirmation of the virtual boundary line and (ii) the modification to the virtual boundary line;
• and
• sending, to the control system, a response to the request, wherein the response comprises the one of (i) the confirmation of the virtual boundary line and (ii) the modification to the virtual boundary line, wherein the control system is configured to control the robotic manipulator to perform the task based on the response.