Robotic manipulator with visual guidance & tactile sensing

A system comprising:

• a sensing device comprising:
• a tactile interface comprising visual markers;
• and
• a camera configured to capture images of the visual markers;
• one or more processors;
• one or more memories having instructions that when executed by the one or more processors, cause the one or more processors to:
• cause the tactile interface of the sensing device to contact a target surface to cause a deformation of the visual markers;
• cause the camera to capture visual feedback associated with the deformation of the visual markers;
• provide, as input to a machine learning algorithm, the visual feedback of the deformation;
• and
• control an operation of a manipulator device based at least in part on an output from the machine learning algorithm.