User interface and related flow for controlling a robotic arm

A system comprising:

• a user interface (UI);
• one or more processors;
• and
• a computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• displaying, via the UI, a first control input for generating a first path for a robot to traverse over a surface of a target object, wherein the target object comprises a set of real-world targets, wherein the first path comprises a set of virtual points, and wherein each virtual point of the set of virtual points comprises a normal of a set of normals, each normal, corresponding to pose for a robot tool to access a real-world target of the set of real-world targets;
• displaying, via a first pane of the UI, a real-time scanning of the target object by the robot, the real-time scanning displaying a first visual marker indicating that the robot has identified a real-world target in real time;
• generating the first path based on a first selection of the first control input;
• detecting, via the UI, a modification of a virtual point of the first path, wherein the modification comprises adding the virtual point to the virtual points, moving the virtual point, or deleting the virtual point from the set of virtual points;
• displaying, via a second pane of the UI, the modification of the virtual point based on detecting the modification of the virtual point, the second pane distinct from the first pane;
• and
• generating a second path based on a second selection of a second control input to generate the second path for the robot to traverse over the surface of the target object, wherein the second path comprises a set of robot tool poses based on the set of normals.

One or more non-transitory computer-readable media including one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• displaying, via a user interface (UI), a first control input for generating a first path for a robot to traverse over a surface of a target object, object, wherein the target object comprises a set of real-world targets, wherein the first path comprises a set of virtual points, and wherein each virtual point of the set of virtual points comprises a normal of a set of normals, each normal, corresponding to pose for a robot tool to access a real-world target of the set of real-world targets;
• displaying, via a first pane of the UI, a real-time scanning of the target object by the robot, the real-time scanning displaying a first visual marker overlaid on the target object indicating that the robot has identified a real-world target in real time;
• generating the first path based on a first selection of the first control input;
• detecting, via the UIL, a modification of a virtual point of the first path, wherein the modification comprises adding the virtual point to the virtual points, moving the virtual point, or deleting the virtual point from the set of virtual points;
• displaying, via a second pane of the UI, the modification of the virtual point based on detecting the modification of the virtual point, the second pane distinct from the first pane;
• and
• generating a second path based on a second selection of a second control input to generate the second path for the robot to traverse over the surface of the target object, wherein the second path comprises a set of robot tool poses based on the set of normals.