# US 2014/0208274 A1 · Controlling a computing-based device using hand gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0208274 A1 - Document ID: US20140208274A1 - Status: Published application - Kind: A1 - Application number: 13/744,823 - Filed: 2013-01-18 - Published: 2014-07-24 - Earliest claimed: 2013-01-18 - Assignees: Microsoft Corporation - Applicants: Microsoft Corporation - Inventors: Samuel Gavin Smyth; Peter John Ansell; Christopher Jozef O'Prey; Mitchel Alan Goldberg; Jamie Daniel Joseph Shotton; Toby Sharp; Shahram Izadi; Abigail Jane Sellen; Richard Malcolm Banks; Kenton O'Hara; Richard Harry Robert Harper; Eric John Greveson; David Alexander Butler; Stephen E Hodges - Web page: https://patentgrep.com/patent/US20140208274A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)} - G06F 18/231: Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram - G06F 18/24155: {Bayesian classification} - G06V 10/7625: {Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms} - G06V 10/764: using classification, e.g. of video objects - G06V 40/113: {Recognition of static hand signs} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods and system for controlling a computing-based device using both input received from a traditional input device (e.g. keyboard) and hand gestures made on or near a reference object (e.g. keyboard). In some examples, the hand gestures may comprise one or more hand touch gestures and/or one or more hand air gestures. ## Claims ### Claim 1 A method of controlling a computing-based device, the method comprising: receiving input from a traditional input device; receiving an image stream of a reference object; performing gesture recognition on the received image stream to identify hand gestures performed on or near the reference object; and controlling the computing-based device using both the input received from the traditional input device and the identified hand gestures. ### Claim 2 (depends on 1) The method according to claim 1, wherein the hand gestures comprise at least one touch hand gesture. ### Claim 3 (depends on 1) The method according to claim 1, wherein the hand gestures comprise at least one free-air hand gesture. ### Claim 4 (depends on 1) The method according to claim 1, wherein the hand gestures comprise at least one touch hand gesture and at least one free-air hand gesture. ### Claim 5 (depends on 1) The method according to claim 1, wherein the reference object is a keyboard. ### Claim 6 (depends on 1) The method according to claim 1, wherein performing gesture recognition on the received image stream comprises identifying the background of the received image stream, wherein identifying the background of the received image stream comprises deeming an image element of the received image stream to be part of the background if the object associated therewith does not move a predetermined distance within a predetermined period. ### Claim 7 (depends on 1) The method according to claim 1, wherein performing gesture recognition on the received image stream comprises performing connected component analysis on at least a part of the image stream to identify components that have at least one image element that lies on a front edge of the image stream. ### Claim 8 (depends on 1) The method according to claim 1, wherein performing gesture recognition on the received image stream comprises detecting at least one arm in the received image stream, wherein the at least one arm is detected by identifying an object in the image stream that has at least one image element that lies on a front edge of the image stream and is greater than a predetermined threshold size. ### Claim 9 (depends on 1) The method according to claim 1, wherein performing gesture recognition on the received image stream comprises applying a classifier to at least a portion of the image elements of the received image stream to classify each image element of the portion of image elements as being part of at least one of a particular body part and a particular state. ### Claim 10 (depends on 9) The method according to claim 9, wherein the classifier is trained using images of at least one synthetic hand and at least one real hand. ### Claim 11 (depends on 9) The method according to claim 9, wherein the classifier produces classification data for each image element of the portion of image elements, the classification data indicting the probability that the image element is part of each of a plurality of possible body parts and each of a plurality of possible states. ### Claim 12 (depends on 11) The method according to claim 11, wherein performing gesture recognition on the received image stream further comprises identifying the location of the centre of mass of a particular body part using the classification data. ### Claim 13 (depends on 12) The method according to claim 12, wherein identifying the centre of mass of the body part comprises summing the product of the location of each image element identified as relating to the body part and the probability that the image element is part of the body part divided by the sum of the probabilities that the image element is part of the body part. ### Claim 14 (depends on 11) The method according to claim 11, wherein performing gesture recognition on the received image stream further comprises identifying the location of a user's digit tip, wherein identifying the location of the user's digit tip comprises: determining the centre of mass of the user's hand; determining the centre of mass of the user's wrist; establishing a virtual line between the centre of mass of the hand and the centre of mass of the wrist; and deeming the user's digit tip to be the image element on the virtual line that is furthest away from the centre of mass of the wrist. ### Claim 15 (depends on 1) The method according to claim 1, wherein performing gesture recognition on the received image stream comprises identifying the plane of a desktop on which the reference object is situated, wherein identifying the plane of the desktop comprises identifying a set of candidate image element likely to correspond to the desktop and iteratively generating a plane and modifying the set of candidate image elements until the plane is a good match for the set of candidate image elements. ### Claim 16 (depends on 1) The method according to claim 1, wherein at least one gesture is identified when a predetermined action is performed in a predetermined location with respect to the reference object. ### Claim 17 (depends on 1) The method according to claim 1, wherein at least one gesture is identified when a user moves a first hand in a predetermined pattern with respect to a second hand. ### Claim 18 A system to control a computing-based device, the system comprising: the computing-based device configured to: receive input from a traditional input device; receive an image stream of a reference object from a capture device; perform gesture recognition on the received image stream to identify hand gestures performed on or near the reference object; and control the computing-based device using both the input received from a traditional input device and the identified hand gestures. ### Claim 19 (depends on 18) The system according to claim 18, the computing-based device being at least partially implemented using hardware logic selected from any one of more of: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device. ### Claim 20 A method of controlling a computing-based device, the method comprising: receiving input from a keyboard; receiving an image stream of the keyboard; performing gesture recognition on the received image stream to identify hand gestures performed on, near or above the keyboard; and controlling the computing-based device using both the input received from the keyboard and the identified hand gestures.