Method and apparatus for integrating manual input

A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:

• two electrical switching means connected together in series having a common node, an input node, and an output node;
• a dielectric-covered sensing electrode connected to the common node between the two switching means;
• a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means;
• and
• an integrating current-to-voltage translation device connected to the output node of the series connected switching means, the current-to-voltage translation device producing a voltage representative of the proximity of the touch device to the sensing device.

A sensing device that is sensitive to changes in self-capacitance brought about by changes in proximity of a touch device to the sensing device, the sensing device comprising:

• two electrical switching means connected together in series having a common node an input node, and an output node,
• a dielectric-covered sensing electrode connected to the common node between the two switching means;
• and
• a power supply providing an approximately constant voltage connected to the input node of the series-connected switching means.

A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

• an array of proximity sensing means embedded in the surface;
• scanning means for forming digital proximity images from the proximities measured by the sensing means;
• image segmentation means for collecting into groups those proximity image pixels intensified by contact of the same distinguishable part of a hand;
• contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
• contact identification means for determining which hand and which part of the hand is causing each surface contact;
• synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time, and for generating command signals in response to synchronous taps of multiple fingers on the surface;
• typing recognition means for generating intended key symbols from asynchronous finger taps;
• motion component extraction means for compressing multiple degrees of freedom of multiple fingers into degrees of freedom common in two and three dimensional graphical manipulation;
• chord motion recognition means for generating one of command and cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers;
• pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted;
• and
• communication means for sending the sensed configurations and activities of finger and palms to one of the electronic or electro-mechanical device.

A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

• an array of proximity sensing means embedded in the surface;
• scanning means for forming digital proximity images from proximities measured by the sensing means;
• contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;
• contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
• contact identification means for determining which hand and which part of the hand is causing each surface contact;
• synchronization detection means for identifying subsets of identified contacts which touchdown or liftoff the surface at approximately the same time;
• typing recognition means for generating intended key symbols from asynchronous finger taps;
• motion component extraction means for compressing the dozens of degrees of freedom in motions of multiple fingers into the degrees of freedom common in two and three dimensional graphical manipulation;
• chord motion recognition means for generating, command or cursor manipulation signals in response to motion in one or more extracted degrees of freedom by a selected combination of fingers;
• and
• communication means for sending said generated input signals to the electronic or electro-mechanical device.

A multi-touch surface apparatus for sensing diverse configurations and activities of fingers and palms of one or more hands near the surface and generating integrated manual input to one of an electronic or electro-mechanical device, the apparatus comprising:

• an array of proximity sensing means embedded in the surface;
• scanning means for forming digital proximity images from proximities measured by the sensing means;
• contact segmentation means for collecting proximity image pixels caused by the same hand part into groups;
• contact tracking means for parameterizing hand contact features and trajectories as the contacts move across successive proximity images;
• contact identification means for determining which hand and which part of the hand is causing each surface contact;
• pen grip detection means for recognizing contact arrangements which resemble the configuration of the hand when gripping a pen, generating inking signals from motions of the inner fingers, and generating cursor manipulation signals from motions of the palms while the inner fingers are lifted;
• and
• communication means for sending said generated input signals to the electronic or electro-mechanical device.

A method for tracking and identifying hand contacts in a sequence of proximity images in order to support interpretation of hand configurations and activities related to typing, multiple degree-of-freedom manipulation via chords, and handwriting, the method comprising the steps of:

• segmenting each proximity image into groups of electrodes which indicate significant proximity, each group representing proximity of a distinguishable hand part or other touch device;
• extracting total proximity, position, shape, size, and orientation parameters from each group of electrodes;
• tracking, group paths through successive proximity images including detection of path endpoints at contact touchdown and liftoff;
• computing velocity and filtered position vectors along each path;
• assigning a hand and finger identity to each contact path by incorporating relative path positions and velocities, individual contact features, and previous estimates of hand and finger positions;
• and
• maintaining estimates of hand and finger positions from trajectories of paths currently assigned to the fingers, wherein the estimates provide high level feedback to bias segmentations and identifications in future images.

A method for filtering and segmenting hand contacts in a sequence of proximity images in order to support interpretation of various contact sizes, shapes, orientations, and spacings, the method comprising the steps of:

• creating a smoothed copy of the most recent proximity image;
• searching for pixels with locally maximum proximity in the smoothed proximity image;
• searching outward from each local maximum pixel for contact boundary pixels using boundary tests of pixel and neighboring pixel proximities which depend on properties of hand contacts expected in a segmentation region of the pixel;
• forming groups from those pixels surrounding each local maximum pixel up to and including the boundary pixels;
• combining groups of pixels which partially overlap;
• extracting group positions and features by fitting an ellipse to each group of pixels;
• and
• updating positions of the segmentation regions of the pixels in response to further analysis of the position and features extracted from each group of pixels.

A method for associating into paths those surface contacts from successive proximity images caused by the same hand part and detecting liftoff from and touchdown onto the surface by each hand part, the method comprising the steps of:

• predicting the current positions of hand parts from their velocity along existing paths;
• finding for each of a group of pixels in current proximity image the existing path with a closest predicted path position;
• finding for each existing path the pixel group whose centroid is closest to the predicted path position and whose centroid is within a path-dependent tracking radius;
• pairing each pixel group with its closest path if the pixel group is also the closest pixel group to the path;
• starting new paths for remaining unpaired pixel groups;
• deactivating paths which have no pairable pixel groups within the path-dependent tracking radius;
• and
• updating path parameters from the measured parameters of the pixel group paired with each path.

A method of computing hand and finger position offsets from the measured positions of individual hand contacts on a multi-touch surface for the purpose of biasing future hand contact identifications or morphing the key layout in an integrated manual input device, the method comprising the steps of:

• establishing fingertip, thumb, or palm identities for each contact;
• establishing an offset weighting for each contact;
• computing a hand position offset, wherein the offset is a weighted average of the difference between a measured position of each contact and a predetermined default position of the hand part which corresponds to an established identity of the contact;
• and
• computing a finger position offset by subtracting a predetermined default position of an associated hand part of the contact and the hand position offset from a measured position of the contact.

A method for establishing identities of hand contacts on a multi-touch surface using relative contact positions and features, the method comprising the steps of:

• defining a template of hand part attractor points on the surface, the attractor points for each hand roughly forming a ring;
• computing a matrix of distances from each surface contact to each attractor point;
• weighting the distances between each surface contact and each attractor point according to how closely measured contact features such as proximity to a surface, shape, size, eccentricity, orientation, distance to nearest neighbor contact, and velocity match features typical of the hand part the attractor point represents;
• finding a one-to-one mapping of the surface contacts to the attractor points that minimizes a sum of distances between each surface contact and its corresponding attractor point;
• and
• recognizing particular hand configurations from the number and features of surface contacts assigned to particular subsets of the attractor points.

A method for ordering surface contacts and establishing finger, thumb, and palm identities, the method comprising the steps of:

• finding a shortest path connecting all of the contacts assumed to be from a given hand;
• passing through each contact once to form an ordered loop;
• finding an innermost contact in the ordered loop;
• determining whether the innermost contact is a thumb, fingertip, or palm contact from contact and inter-contact features of the innermost contact;
• and
• assigning thumb, fingertip, or palm identities to non-innermost contacts based upon the features of the contacts, assignment of the innermost contacts, vertical position relative to assigned contacts, and the loop ordering.

An apparatus for distinguishing palm heel contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:

• means for measuring the total proximity, orientation, and eccentricity of all contacts;
• means for encouraging identification of a given contact as a palm heel contact if its ratio of total proximity to eccentricity is larger than for a typical fingertip contact;
• and
• means for encouraging identification of a given contact as a palm heel contact as its orientation approaches the expected slant of a palm heel.

An apparatus for distinguishing thumb contacts from other types of hand contacts in a system for recognizing hand activity on a multi-touch surface and generating input signals to a competing device therefrom, the apparatus comprising:

• means for measuring the size and orientation of all contacts;
• means for encouraging identification of a given contact as a thumb contact if its size is larger than a typical fingertip contact;
• means for discouraging identification of a given contact as a thumb contact if its size is larger than a typical thumb contact;
• and
• means for encouraging identification of a given contact as a thumb contact as its orientation approaches the expected slant of the thumb.

A method for determining which hand causes each surface contact detected on a multi-touch surface so that input signals generated by hand activity on the surface can depend on the identity of the hand performing the activity and so that multiple hands can perform independent activities on the surface simultaneously, the method comprising the steps of:

• defining a template of hand part attractor points on the surface, the attractor points for each hand approximately forming a ring;
• generating partitions which divide the set of all surface contacts into left hand clusters and right hand clusters;
• assigning finger and palm identities to the contacts within each cluster;
• computing for each partition an assignment fitness measure which represents the biomechanical consistency of the fit of contact clusters to their assigned attractor rings;
• choosing the partition which has the best assignment fitness measure as the partition containing the true contact identities;
• and
• recognizing each hand's configuration from the combination of and features of surface contacts assigned within each attractor ring of the best partition.

A method for integrally extracting multiple degrees of freedom of hand motion from sliding motions of two or more fingers of a hand across a multi-touch surface, one of the fingers preferably being the opposable thumb, the method comprising the steps of:

• tracking, across successive scans of the proximity sensor array the trajectories of individual hand parts on the surface;
• finding an innermost and an outermost finger contact from contacts identified as fingers on the given hand;
• computing a scaling velocity component from a change in a distance between the innermost and outermost finger contacts;
• computing a rotational velocity component from a change in a vector angle between the innermost and outermost finger contacts;
• computing a translation weighting for each contacting finger;
• computing translational velocity components in two dimensions from a translation weighted average of the finger velocities tangential to surface;
• suppressively filtering components whose speeds are consistently lower than the fastest components;
• transmitting the filtered velocity components as control signals to an electronic or electro-mechanical device.

A method for integrally extracting roll and tilt degrees of freedom of hand motion from pressure changes of three or more non-collinear hand contacts comprising any of thumbs, fingertips or palms, the method comprising the steps of:

• tracking across successive proximity images the trajectories of individual hand parts on the surface;
• measuring proximities from each hand contact in a calibration proximity image once all available hand contacts have been stabilized;
• computing an average hand contact position from a post-calibration proximity image, wherein all hand contacts are weighted equally;
• computing a weighted average hand contact position from a post-calibration proximity image, wherein each hand contact is weighted according to the ratio of its current proximity to its calibrated proximity;
• computing for each post-calibration proximity image the difference vector between the weighted average hand contact position and the average hand contact position;
• dead-zone filtering the difference vector to remove variations in proximity due to unintentional posture shifts;
• and
• transmitting the filtered difference vector from each post-calibration proximity image as roll and tilt control signals to an electronic or electro-mechanical device.

A manual input integration method for supporting diverse hand input activities such as resting the hands, typing, multiple degree-of-freedom manipulation, command gesturing and handwriting on a multi-touch surface, the method enabling users to instantaneously switch between the input activities by placing their hands in different configurations comprising distinguishable combinations of relative hand contact timing, proximity, shape, size, position, motion and/or identity across a succession of surface proximity images, the method comprising the steps of:

• tracking each touching hand part across successive proximity images;
• measuring the times when each hand part touches down and lifts off the surface;
• detecting when hand parts touch down or lift off simultaneously;
• producing discrete key symbols when the user asynchronously taps, holds, or slides a finger on key regions defined on the surface;
• producing discrete mouse button click commands, key commands, or no signals when the user synchronously taps two or more fingers from the same hand on the surface;
• producing gesture commands or multiple degree-of-freedom manipulation signals when the user slides two or more fingers across the surface;
• and
• sending the produced symbols, commands and manipulation signals as input to an electronic or an electro-mechanical device.

A method for choosing what kinds of input signals will be generated and sent to an electronic or electro-mechanical device in response to tapping or sliding of fingers on a multi-touch surface, the method comprising the following steps:

• identifying each contact on the surface as either a thumb, fingertip or palm;
• measuring the times when each hand part touches down and lifts off the surface;
• forming a set of those fingers which touch down from the all finger floating state before any one of the fingers lifts back off the surface;
• choosing the kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the set;
• generating input signals of this kind when further distinctive motions of the fingers occur;
• forming a subset any two or more fingers which touch down synchronously after at least one finger has lifted back off the surface;
• choosing a new kinds of input signals to be generated by further distinctive motion of the fingers from the combination of finger identities in the subset;
• generating input signals of this new kind when further distinctive motions of the fingers occur;
• and
• continuing to form new subsets, choose and generate new kinds of input signals in response to liftoff and synchronous touchdowns until all fingers lift off the surface.

A method for continuing generation of cursor movement or scrolling signals from a tangential motion of a touch device over a touch-sensitive input device surface after touch device liftoff from the surface if the touch device operator indicates that cursor movement continuation is desired by accelerating or failing to decelerate the tangential motion of the touch device before the touch device is lifted, the method comprising the following steps:

• measuring, storing and transmitting to a computing device two or more representative tangential velocities during touch device manipulation;
• computing and storing a liftoff velocity from touch device positions immediately prior to the touch device liftoff;
• comparing the liftoff velocity with the representative tangential velocities, and entering a mode for continuously moving the cursor if a tangential liftoff direction approximately equals the representative tangential directions and a tangential liftoff speed is greater than a predetermined fractional multiple of representative tangential speeds;
• continuously transmitting cursor movement signals after liftoff to a computing device such that the cursor movement velocity corresponds to one of the representative tangential velocities;
• and
• ceasing transmission of the cursor movement signals when the touch device engages the surface again, if comparing means detects significant deceleration before liftoff, or if the computing device replies that the cursor can move no farther or a window can scroll no farther.

A method for mapping gestures performed on a multi-touch surface by a right hand to simulate mouse manipulations, the method comprising the steps of:

• reserving asynchronous single-finger motions for typing and five-finger motions for hand resting;
• generating mouse pointer motion signals in response to translational slides of two fingertips;
• generating a single mouse click signal in response to a synchronized tap of two fingertips;
• generating mouse drag signals in response to translational slides of three fingertips;
• generating a double mouse click signal in response to a synchronized tap of three fingertips;
• generating window scrolling signals in response to translational slides of four fingertips;
• generating a cut to clipboard signal in response to a pinching motion between the thumb and a fingertip;
• generating a copy to clipboard signal in response to a synchronized tap of the thumb and a fingertip;
• and
• generating a paste from clipboard signal in response to a movement of the thumb and a fingertip away from each other.