# US 2014/0192024 A1 · Object detection and tracking with audio and optical signals > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0192024 A1 - Document ID: US20140192024A1 - Status: Published application - Kind: A1 - Application number: 14/149,709 - Filed: 2014-01-07 - Published: 2014-07-10 - Earliest claimed: 2013-01-08 - Assignees: Leap Motion, Inc. - Applicants: Leap Motion, Inc. - Inventors: David HOLZ - Web page: https://patentgrep.com/patent/US20140192024A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04166: {Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving (Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184)} - G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/043: using propagating acoustic waves - G06F 2203/04106: Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The technology disclosed addresses these problems by supplementing optical gesture recognition with the ability to recognize touch gestures. This capability allows the user to execute intuitive gestures involving contact with a surface. For example, in low-light situations where free-form gestures cannot be recognized optically with a sufficient degree of reliability, a device implementing the invention may switch to a touch mode in which touch gestures are recognized. In implementations, two contact microphones or other vibratory or acoustical sensors, that are coupled to an optical motion-capture and are in contact with a surface that a user touches, are monitored. When the contact microphones detect audio signals (or other vibrational phenomena) generated by contact of an object with the surface that the user touches, a position of the object traveling across and in contact with the surface is tracked. ## Claims ### Claim 1 A method of operating an optical motion-capture system in low-light situations, the method including: monitoring at least two contact microphones coupled to an optical motion-capture system, wherein the contact microphones are in contact with a surface that a user touches; and in response to detection of audio signals generated by contact of an object with the surface that the user touches, tracking a position of the object by: measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA; analyzing at least one image, captured by a camera of the optical motion-capture system, of the object in contact with the surface; and using the image analysis to select among the mapped TDOA surface locations as a surface location of the contact. ### Claim 2 (depends on 1) The method of claim 1, further including calculating a path of the object traveling across and in contact with the surface by: repeatedly measuring TDOAs of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and analyzing a sequence of images of the successive contacts of the object with the surface. ### Claim 3 (depends on 2) The method of claim 2, further including comparing the path to a plurality of path templates and identifying a template that best matches the path. ### Claim 4 (depends on 1) The method of claim 1, further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals. ### Claim 5 (depends on 1) The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object. ### Claim 6 (depends on 1) The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the tracked object. ### Claim 7 A method of tracking a path of an object of interest traveling across and in contact with a surface, the method comprising: capturing, with a camera, a sequence of images of a field of view that includes at least a portion of the surface; capturing audio signals through the surface using at least two microphones; and computationally constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals. ### Claim 8 (depends on 7) The method of claim 7, wherein the 3D model includes a position and shape of the object. ### Claim 9 (depends on 7) The method of claim 7, wherein tracking movement includes calculating a location and direction of the object by computationally analyzing the audio signals. ### Claim 10 (depends on 9) The method of claim 9, wherein the location and direction of the object is calculated using a time difference of arrival. ### Claim 11 (depends on 7) The method of claim 7, further including filtering the audio signals to exclude frequencies outside a defined range. ### Claim 12 (depends on 7) The method of claim 7, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory or a gesture of the tracked object. ### Claim 13 (depends on 12) The method of claim 12, wherein the trajectory or the gesture dictates a change in operational mode. ### Claim 14 An optical motion-capture system, including: a camera for capturing images of an object within a field of view thereof; a contact microphone coupled to the camera and in contact with a surface that a user touches; a sensor for monitoring the contact microphone; a control module for tracking a position of the object, in response to detection of audio signals generated by contact of an object with the surface that the user touches, by: measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA; analyzing at least one image, captured by the camera of the optical motion-capture system, of the object in contact with the surface; and using the image analysis to select among the mapped TDOA surface locations as a surface location of the contact. ### Claim 15 (depends on 14) The system of claim 14, further configured to calculate a path of the object traveling across and in contact with the surface by: repeatedly measuring TDOAs of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and analyzing a sequence of images of the successive contacts of the object with the surface. ### Claim 16 (depends on 14) The system of claim 14, further configured to construct a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals. ### Claim 17 (depends on 14) The system of claim 14, further configured to recognize at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object. ### Claim 18 (depends on 14) The system of claim 14, further configured to recognize at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.