# US 2013/0155026 A1 · New kind of multi-touch input device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0155026 A1 - Document ID: US20130155026A1 - Status: Published application - Kind: A1 - Application number: 13/706,761 - Filed: 2012-12-06 - Published: 2013-06-20 - Earliest claimed: 2011-12-16 - Assignees: International Business Machines Corporation - Inventors: Jin Dong; Yu Hui Fu; Qi Ming Tian; Ming Xie; Wen Jun Yin; Wei Zhao - Web page: https://patentgrep.com/patent/US20130155026A1 ## CPC - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A multi-touch method includes, in response to at least one finger of a user pointing toward a computer screen, recognizing an eye and at least one finger in an image frame taken by a camera; mapping a first coordinate of the recognized finger in the image frame to a second coordinate of the finger on the computer screen; mapping a third coordinate the recognized eye in the image frame to a fourth coordinate of the eye on the computer screen; determining whether the fourth coordinate is in the proximity of the second coordinate; in response to a positive decision, determining the second coordinate as a multi-touch point. ## Claims ### Claim 1 A multi-touch method, comprising: in response to at least one finger of a user pointing toward a computer screen, recognizing an eye and at least one finger in an image frame taken by a camera; mapping a first coordinate of the recognized finger in the image frame to a second coordinate of the finger on the computer screen; mapping a third coordinate of the recognized eye in the image frame to a fourth coordinate of the eye on the computer screen; determining whether the fourth coordinate is in the proximity of the second coordinate; in response to a positive decision, determining the second coordinate as a multi-touch point. ### Claim 2 (depends on 1) The method according to claim 1, further comprising: assigning an unique ID for a multi-touch point corresponding to the second coordinate; sending the ID and the second coordinate as parameters of a multi-touch event to an operating system. ### Claim 3 (depends on 2) The method according to claim 2, further comprising: in response to recognizing that an adjacent image frame of the image frame contains a multi-touch point, determining whether a multi-touch point contained image sub-block in the adjacent image frame matches a multi-touch point contained image sub-block in the image frame; in response to a positive determining result, identifying the multi-touch point contained in the adjacent image frame with the ID. ### Claim 4 (depends on 1) The method according to claim 1, wherein the fourth coordinate of the eye on the computer screen is calculated based on the following parameters: a third coordinate of the eye in the image frame, the size α of the eye, a relationship coefficient n between the size α of the eye and a distance d between the eye and the computer screen, and the orientation of the pupil in the spatial coordinate system. ### Claim 5 (depends on 4) The method according to claim 4, further comprising: selecting sampled finger and eye images as positive samples; selecting arbitrary images as negative samples; training a classifier with the positive and negative samples. ### Claim 6 (depends on 5) The method according to claim 5, wherein recognizing an eye and at least one finger in an image frame taken by a camera comprises: partitioning a collected image frame to a plurality of image sub-blocks; recognizing an eye and at least one finger in the plurality of image sub-blocks with the classifier. ### Claim 7 (depends on 6) The method according to claim 6, further comprising: extracting the boundary of the recognized finger and the boundary of the recognized eyeball through an image boundary detection and extraction algorithm; calculating the first coordinate according to the number and coordinates of pixels on the boundary of the finger; calculating the third coordinate according to the number and coordinates of pixels on the boundary of the eyeball. ### Claim 8 A multi-touch apparatus, comprising: a recognition module, configured to, in response to at least one finger of a user pointing toward a computer screen, recognize an eye and at least one finger in an image frame taken by a camera; a coordinate mapping module, configured to map a first coordinate of the recognized finger in the image frame to a second coordinate of the finger on the computer screen; and map a third coordinate of the recognized eye in the image frame to a fourth coordinate of the eye on the computer screen; a determination module, configured to determine whether the fourth coordinate is in the proximity of the second coordinate; a multi-touch point determination module, configured to, in response to a positive decision, determine the second coordinate as a multi-touch point. ### Claim 9 (depends on 8) The apparatus according to claim 8, further comprising: an ID assignment module, configured to assign a unique ID for a multi-touch point corresponding to the second coordinate; a parameter transmission module, configured to send the ID and the second coordinate as parameters of a multi-touch event to an operating system. ### Claim 10 (depends on 9) The apparatus according to claim 9, wherein the ID assignment module is further configured to: in response to recognize that an adjacent image frame of the image frame contains a multi-touch point, determine whether a multi-touch point contained image sub-block in the adjacent image frame matches a multi-touch point contained image sub-block in that image frame; in response to a positive determining result, identify the multi-touch point contained in the adjacent image frame with the ID. ### Claim 11 (depends on 8) The apparatus according to claim 8, wherein the fourth coordinate of the eye on the computer screen is calculated based on the following parameters: a third coordinate of the eye in the image frame, the size α of the eye, a relationship coefficient n between the size α of the eye and a distance d between the eye and the computer screen, and the orientation of the pupil of the eye in the spatial coordinate system. ### Claim 12 (depends on 11) The apparatus according to claim 11, further comprising a classifier training module configured to: select sampled finger and eye images as positive samples; select arbitrary images as negative samples; train a classifier with the positive and negative samples. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein the recognition module is further configured to: partition a collected image frame to a plurality of image sub-blocks; recognize an eye and at least one finger in the plurality of image sub-blocks with the classifier. ### Claim 14 (depends on 13) The apparatus according to claim 13, further comprising an image coordinate calculation module, configured to: extract the boundary of the recognized finger and the boundary of the recognized eyeball through an image boundary detection and extraction algorithm; calculate the first coordinate according to the number and coordinates of pixels on the boundary of the finger; calculate the third coordinate according to the number and coordinates of pixels on the boundary of the eyeball. ### Claim 15 (depends on 8) A portable terminal device, comprising a multi-touch apparatus according to claim 8.