Method and system for providing remote robotic control

A computing device for providing mixed-initiative robotic control, including:

• one or more processors;
• and
• memory storing instructions, wherein:
• the instructions, when executed by the one or more processors, cause the processors, to perform operations comprising:
• displaying a control user interface via a display generation component in communication with the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding a robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot, wherein the virtualized environment includes a first virtual object located at a first location within the virtualized environment, and wherein the robot executes a first navigation path in the physical environment that is generated in accordance with a first set of preprogrammed path-planning instructions and the first location of the first virtual object;
• while the robot is executing the first navigation path in the physical environment, detecting a first user input, including detecting a first movement input directed to the first virtual object;
• and
• in response to detecting the first user input:
• moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions, and wherein:
• the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object, or,
• the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object.

A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors of a computing device, cause the computing device to perform operations, wherein:

• the operations include:
• displaying a control user interface via a display generation component in communication with the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding the robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot, wherein the virtualized environment includes a first virtual object located at a first location within the virtualized environment, and wherein the robot executes a first navigation path in the physical environment that is generated in accordance with a first set of preprogrammed path-planning instructions and the first location of the first virtual object;
• while the robot is executing the first navigation path in the physical environment, detecting a first user input, including detecting a first movement input directed to the first virtual object;
• and
• in response to detecting the first user input:
• moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions, and wherein:
• the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object, or,
• the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object.