Device and method to identify functional parts of tools for robotic manipulation

A computer-implemented method for identifying functional elements of an object for robotic manipulation, comprising an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of:

• causing a robot to repeatedly manipulate an object, wherein the robot comprises an audio sensor and a plurality of touch sensors;
• recording a three-dimensional (3D) location of each audio event that produces a response as measured by the audio sensor during manipulation of the object;
• recording a 3D location of each tactile event that produces a response as detected by the plurality of touch sensors during manipulation of the object;
• determining a 3D location of each audio and tactile event in 3D space;
• generating a 3D audio point cloud denoting the 3D locations of the audio events;
• generating a 3D tactile point cloud denoting the 3D locations of the tactile events;
• registering the 3D audio point cloud and the 3D tactile point cloud with a 3D model point cloud of the object;
• and
• annotating the 3D model point cloud of the object with the audio events to generate an annotated 3D model point cloud of the object, which encodes the location of a functional element of the object.

A computer program product for identifying functional elements of an object for robotic manipulation, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

• causing a robot to repeatedly manipulate an object, wherein the robot comprises an audio sensor and a plurality of touch sensors;
• recording a three-dimensional (3D) location of each audio event that produces a response as measured by the audio sensor during manipulation of the object;
• recording a 3D location of each tactile event that produces a response as detected by the plurality of touch sensors during manipulation of the object;
• determining a 3D location of each audio and tactile event in 3D space;
• generating a 3D audio point cloud denoting the 3D locations of the audio events;
• generating a 3D tactile point cloud denoting the 3D locations of the tactile events;
• registering the 3D audio point cloud and the 3D tactile point cloud with a 3D model point cloud of the object;
• and
• annotating the 3D model point cloud of the object with the audio events to generate an annotated 3D model point cloud of the object, which encodes the location of a functional element of the object.