Recognizing gestures and using gestures for interacting with software applications

A computer readable storage medium storing machine instructions for performing a method for using a gesture in connection with an interactive display system on which images are displayed and objects disposed on or proximate thereto are optically detectable, the method comprising:

• (a) processing an image to distinguish between touch objects physically touching a display surface and hover objects proximate to the display surface and to recognize a gesture that was made by the a user using the object, said gesture being recognized in response to detecting a plurality of touch connected components and at least one hover connected component to generally determine an orientation of an appendage of the user, said connected components comprising adjacent pixels having an intensity above a predefined threshold;
• and
• (b) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture, said response changing a state of the interactive display system;
• (c) analyzing temporal patterns of a plurality of sets of touch connected components as the touch connected components change over time;
• (d) identifying appendages of each of a plurality of different users, where the appendages are hovering adjacent to the display surface;
• and
• (e) determining that groupings of multiple fingers in contact with the display surface belong to specific ones of the appendages, and thus, belong to specific users, based on an orientation of each appendage, and on the temporal patterns.

An interactive display system that responds to a gesture made with an object, comprising:

• (a) a display surface having an interactive side on or adjacent to which an object can be placed and manipulated, and an opposite side;
• (b) a light source that directs light toward the opposite side of the display surface and through the display surface, to the interactive side;
• (c) a light sensor disposed to receive and sense light reflected back from an object through the display surface, producing a signal corresponding to an image of an object that is in contact or proximate to the display surface;
• (d) a processor in communication with the light sensor;
• and
• (e) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including:
• (i) processing the image to distinguish between touch objects physically touching the display surface and hover objects proximate to the display surface and to recognize a gesture that was made by the user using the object, said gesture being recognized in response to detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity above a predefined threshold;
• and
• (ii) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture in the memory, said response changing a state of the interactive display system.