# US 2013/0155070 A1 · Method for user input from alternative touchpads of a handheld computerized device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0155070 A1 - Document ID: US20130155070A1 - Status: Published application - Kind: A1 - Application number: 13/770,791 - Filed: 2013-02-19 - Published: 2013-06-20 - Earliest claimed: 2010-04-23 - Assignees: Handscape Inc. - Inventors: Tong Luo - Web page: https://patentgrep.com/patent/US20130155070A1 ## CPC - G06T 11/20: Drawing from basic elements - G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]} - G06F 1/1643: {the display being associated to a digitizer, e.g. laptops that can be used as penpads (details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615)} - G06F 1/169: {the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes (G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033)} - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present invention relates to a handheld computerized device with a bit mapped display screen on the front panel, and a touchpad installed in an alternative location. More particularly, the invention relates to a method and graphical user interface that can, for example, enable the user to see the user's finger position and motion from behind the device superimposed upon a virtual keyboard layout on the front panel. This can allow the user to use a touchpad keypad on the back of the device to input keystrokes and mouse actions, and these will be reflected on the display screen on the front of the handheld computerized device as “virtual fingers” or equivalent. The system operates by inputting touchpad data into a software biomechanical and anatomical model of the human hand, optionally supplemented by video or image information to refine this model. ## Claims ### Claim 1 A method of creating a virtual image of at least portions of the user's hand or hands while operating a handheld computerized device, said handheld computerized device comprising at least one touchpad, at least one graphics display screen, at least one processor, memory, and software, said method comprising: obtaining data on the location and movement of the user's fingers and/or hand using said touchpad; said touchpad being located in a location of said handheld computerized device that is different from the location of said at least one display screen; analyzing said data from said touchpad on the location and movement of said user's fingers and/or hand according to a biomechanical and anatomical model of a human hand, and assigning said data on the location and movement of said user's fingers and/or hand to specific fingers on said biomechanical and anatomical model of said human hand, thereby making predictions as to the location of the user's hand and fingers; using said biomechanical and anatomical model of said human hand, and said predictions of the location of the user's hand and fingers, to compute a graphical representation of at least said user's fingers; displaying said graphical representation of at least said user's fingers on said at least one graphics display screen of said handheld computerized device. ### Claim 2 (depends on 1) The method of claim 1, wherein the biomechanical and anatomical model of a human hand is supplemented or refined by obtaining at least one full or partial image of the user hand or hands. ### Claim 3 (depends on 2) The method of claim 2, further analyzing said at least one full or partial image of the user's hand or hands to compute one or more user parameters of said biomechanical and anatomical model of said human hand; and using said one or more user parameters of said biomechanical and anatomical model of said human hand to replace one or more default parameters of said biomechanical and anatomical model of said human hand. ### Claim 4 (depends on 3) The method of claim 3, wherein said one or more user parameters comprise the length of at least one or more distal phalanges or intermediate phalanges or proximate phalanges or metacarpals of said user's hand or hands. ### Claim 5 (depends on 2) The method of claim 2, wherein said handheld computerized device additionally comprises at least one video camera configured to photograph at least part of the region of space proximate said at least one touchpad, but not configured to photograph the surface of said touchpad; wherein the position of at least the distal phalanges of said user's fingers are not photographed by said at least one video camera when said user is touching said touchpad; wherein the biomechanical and anatomical model of a human hand is further supplemented or refined by real-time video images of at least non-finger regions of said human hand obtained from said at least one camera. ### Claim 6 (depends on 1) The method of claim 1, wherein said touchpad is located on a side of said handheld computerized device that is different from the side of said handheld device that holds said at least one graphics display screen. ### Claim 7 (depends on 1) The method of claim 1, wherein said touchpad is capable of detecting the proximity of said user's fingers above the surface of said touchpad; wherein said touchpad is capable of distinguishing the distance between said user's fingers and the surface of said touchpad when said distance is greater than zero. ### Claim 8 (depends on 1) The method of claim 1, in which said software comprises calibration software to enable the biomechanical and anatomical model of said human hand to be calibrated to the touchpad data to more accurately match the biomechanical and anatomical characteristics of said user's fingers and/or hand; and wherein said biomedical and anatomical characteristics comprise at least the length of the user's fingers, thereby allowing said model of said human hand to determine the length of said user's fingers for a plurality of different hand configurations. ### Claim 9 (depends on 1) The method of claim 1, used to assist user data entry into said handheld computerized device, further comprising: displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device; wherein said at least one data entry location is a graphical display of a keyboard or keypad comprised of a plurality of data entry locations, or wherein said at least one data entry location is a hyperlink. ### Claim 10 (depends on 9) The method of claim 9, in which said at least one data entry location is highlighted on said at least one graphics display screen whenever said computerized device determines that at least one finger on said user's hand has left the touchpad and the position and motion history of said at least one finger is consistent with a capability said at least one finger on said user's hand to strike a position on said touchpad that is consistent with the location of said at least one data entry location on said at least one graphics display screen; wherein said highlighting commences before said at least one finger has struck the next position on said keypad. ### Claim 11 (depends on 9) The method of claim 9, in which said at least one data entry location comprises alphanumeric letter data entry locations, and wherein said software comprises predictive typing software configured to deduce the most probable letter or letters that the user is likely to type next, and the most probable finger or fingers that the user will use to type this most probable letter, and said computerized device utilizes keystroke predictions made by said predictive typing software, along with said data on the location and movement of said user's fingers and/or hand using said touchpad, to compute the graphical representation of at least said user's fingers. ### Claim 12 (depends on 9) The method of claim 9 wherein either the graphical representation of at least said user's fingers on said at least one graphics display screen is semi-transparent so as to enable the graphical display and said user's fingers to be simultaneously observed; or wherein said at least one data entry location is a graphical display of a semi-transparent keyboard or semi-transparent keypad, thus allowing details of a current application running on said graphics display to be visualized along with said semi-transparent keyboard or semi-transparent keypad. ### Claim 13 A method of creating a virtual image of at least portions of the user's hand or hands while operating a handheld computerized device, said handheld computerized device comprising at least one touchpad, at least one graphics display screen, at least one processor, memory, and software, said method comprising: further displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device; obtaining data on the location and movement of the user's fingers and/or hand using said touchpad; said touchpad being located in a location of said handheld computerized device that is different from the location of said at least one display screen; analyzing said data from said touchpad on the location and movement of said user's fingers and/or hand according to a biomechanical and anatomical model of a human hand, assigning said data on the location and movement of said user's fingers and/or hand to specific fingers on said biomechanical and anatomical model of said human hand, thereby making predictions as to the location of the user's hand and fingers; wherein the biomechanical and anatomical model of a human hand is supplemented or refined by obtaining at least one full or partial image of the user hand or hands; using said biomechanical and anatomical model of said human hand, and said predictions of the location of the user's hand and fingers, to compute a graphical representation of at least said user's fingers; displaying said graphical representation of at least said user's fingers on said at least one graphics display screen of said handheld computerized device. wherein distances between said graphical representation of at least said user's fingers on said at least one graphics display screen, and said at least one data entry location, give information to said user to facilitate said user to position said user's fingers and/or hand on said at least one touchpad to enter user data into said at least one data entry location. ### Claim 14 (depends on 13) The method of claim 13, further analyzing said at least one full or partial image of the user's hand or hands to compute one or more user parameters of said biomechanical and anatomical model of said human hand; and using said one or more user parameters of said biomechanical and anatomical model of said human hand to replace one or more default parameters of said biomechanical and anatomical model of said human hand; wherein said one or more user parameters comprise the length of at least one or more distal phalanges or intermediate phalanges or proximate phalanges or metacarpals of said user's hand or hands. ### Claim 15 (depends on 13) The method of claim 13, wherein said handheld computerized device additionally comprises at least one video camera configured to photograph at least part of the region of space proximate said at least one touchpad, but not configured to photograph the surface of said touchpad; wherein the position of at least the distal phalanges of said user's fingers are not photographed by said at least one video camera when said user is touching said touchpad; wherein the biomechanical and anatomical model of a human hand is further supplemented or refined by real-time video images of at least non-finger regions of said human hand obtained from said at least one camera. ### Claim 16 (depends on 13) The method of claim 13, in which said software comprises calibration software to enable the biomechanical and anatomical model of said human hand to be calibrated to the touchpad data to more accurately match the biomechanical and anatomical characteristics of said user's fingers and/or hand; and wherein said biomedical and anatomical characteristics comprise at least the length of the user's fingers, thereby allowing said model of said human hand to determine the length of said user's fingers for a plurality of different hand configurations. ### Claim 17 (depends on 13) The method of claim 13, used to assist user data entry into said handheld computerized device, further comprising: displaying at least one data entry location on said at least one graphics display screen of said handheld computerized device; wherein said at least one data entry location is a graphical display of a keyboard or keypad comprised of a plurality of data entry locations, or wherein said at least one data entry location is a hyperlink. ### Claim 18 (depends on 17) The method of claim 17, in which said at least one data entry location is highlighted on said at least one graphics display screen whenever said computerized device determines that at least one finger on said user's hand has left the touchpad and the position and motion history of said at least one finger is consistent with a capability said at least one finger on said user's hand to strike a position on said touchpad that is consistent with the location of said at least one data entry location on said at least one graphics display screen; wherein said highlighting commences before said at least one finger has struck the next position on said keypad. ### Claim 19 (depends on 17) The method of claim 17, in which said at least one data entry location comprises alphanumeric letter data entry locations, and wherein said software comprises predictive typing software configured to deduce the most probable letter or letters that the user is likely to type next, and the most probable finger or fingers that the user will use to type this most probable letter, and said computerized device utilizes keystroke predictions made by said predictive typing software, along with said data on the location and movement of said user's fingers and/or hand using said touchpad, to compute the graphical representation of at least said user's fingers. ### Claim 20 (depends on 17) The method of claim 17 wherein either the graphical representation of at least said user's fingers on said at least one graphics display screen is semi-transparent so as to enable the graphical display and said user's fingers to be simultaneously observed; or wherein said at least one data entry location is a graphical display of a semi-transparent keyboard or semi-transparent keypad, thus allowing details of a current application running on said graphics display to be visualized along with said semi-transparent keyboard or semi-transparent keypad.