Object detection and tracking with audio and optical signals

A method of tracking a path of an object of interest traveling across and in contact with a surface, the method comprising:

• capturing, with a camera, a sequence of images of a field of view that includes at least a portion of the surface;
• capturing audio signals through the surface using at least two microphones;
• and
• computationally constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

An optical motion-capture system, including:

• a camera for capturing images of an object within a field of view thereof;
• a contact microphone coupled to the camera and in contact with a surface that a user touches;
• a sensor for monitoring the contact microphone;
• a control module for tracking a position of the object, in response to detection of audio signals generated by contact of an object with the surface that the user touches, by:
• measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA;
• analyzing at least one image, captured by the camera of the optical motion-capture system, of the object in contact with the surface;
• and
• using the image analysis to select among the mapped TDOA surface locations as a surface location of the contact.