Methods and systems for six-degree-of-freedom haptic interaction with streaming point data

A non-transitory computer readable storage medium storing instructions that, upon execution by one or more processors, cause the one or more processors to implement a method comprising:

• receiving sensor data corresponding to a remote environment, wherein the sensor data includes information corresponding to a target object;
• generating a point cloud based on the sensor data, wherein the point cloud includes points generated via the information corresponding to the target object;
• determining that the target object is a recognized object;
• based on the determination that the target object is a recognized object, generating a first virtual fixture associated with the target object and a first task;
• generating a second virtual fixture associated with a second task;
• monitoring movement of a manipulator in the remote environment via the sensor data to determine a completion of the first task, wherein the second virtual fixture is generated in response to the completion of the first task;
• generating a virtual environment based on the point cloud, wherein the virtual environment includes the first virtual fixture and the second virtual fixture;
• rendering at least a portion of the virtual environment on a display;
• and
• providing haptic feedback via a control mechanism, wherein the haptic feedback is associated with movement of the manipulator, and wherein the haptic feedback includes first feedback associated with the first virtual fixture and second feedback associated with the second virtual fixture.

A non-transitory computer readable storage medium storing instructions that, upon execution by one or more processors, cause the one or more processors to implement a method comprising:

• collecting sensor data in an environment, wherein the sensor data includes
• first information corresponding to the environment, and second information corresponding to a target object in the environment, wherein the first information and the second information include depth data;
• assigning a label to the second information, wherein the label associates the second information with the target object;
• generating a virtual environment based on the sensor data, wherein the virtual environment includes a virtual object associated with the labeled second information;
• generating a first virtual fixture associated with the virtual object and a first task;
• transmitting signals from a control mechanism located remotely from the environment to a manipulator in the environment;
• moving the manipulator to manipulate the target object;
• providing first haptic feedback through the control mechanism via the first virtual fixture;
• monitoring movement of the manipulator via the sensor data to determine a completion of the first task;
• upon completion of the first task, generating a second virtual fixture associated with a second task;
• and
• providing second haptic feedback through the control mechanism via the second virtual fixture.