Virtual presence for telerobotics in a dynamic scene

A computerized method for providing virtual presence for telerobotics in a dynamic scene, the method comprising:

• capturing, by a sensor device, one or more frames of a scene comprising one or more objects, each frame comprising (i) one or more color images of the scene and the one or more objects and (ii) one or more depth maps of the scene and the one or more objects;
• receiving, by a computing device coupled to the sensor device, a first set of frames from the sensor device;
• for each frame in the first set of frames:
• generating, by the computing device coupled to the sensor device, a set of feature points corresponding to the one or more objects in the scene,
• matching, by the computing device, the set of feature points to one or more corresponding 3D points in a depth map from the frame, and
• constructing, by the computing device, a dense mesh of the scene and the one or more objects using the matched feature points;
• receiving, by the computing device, a second set of frames from the sensor device;
• for each frame in the second set of frames:
• a) calculating, by the computing device, a first geometric error between one or more segments of the dense mesh and corresponding 3D points in a depth map from the frame,
• b) classifying, by the computing device, one or more segments of the dense mesh as one or more dynamic segments based on the first geometric error calculated for the one or more segments,
• c) non-rigidly deforming, by the computing device, only the one or more segments of the dense mesh that are classified as the one or more dynamic segments using data associated with the corresponding 3D points in the depth map from the frame,
• d) calculate a second geometric error between the deformed one or more dynamic segments and corresponding 3D points in the depth map from the frame,
• e) modify the deformed dense mesh using a non-linear optimization if the second geometric error does not converge,
• f) repeating, by the computing device, steps d) and e) until the second geometric error converges, and
• g) updating, by the computing device, a texture of the deformed dense mesh by aligning a color image from the frame to the deformed dense mesh;
• determining, by the computing device, net changes to the dense mesh resulting from the non-rigid deformation and transmitting (i) the net changes to the dense mesh of the scene and the one or more objects and (ii) the frame to a remote viewing device, the remote viewing device coupled to a remote controller;
• updating, by the remote viewing device, an existing 3D representation of the scene and the one or more objects using the net changes to the dense mesh and the frame for display to a user;
• receiving, by the remote viewing device, one or more commands from the user via the remote controller, the one or more commands corresponding to interaction with the one or more objects in the updated 3D representation of the scene;
• transmitting, by the remote viewing device, the one or more commands to a robot device that interacts with the one or more objects in the scene;
• and
• executing, by the robot device, the one or more commands received from the remote viewing device to perform one or more operations.