# US 11,029,845 B2 · Virtual keyboard engagement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,845 B2 - Document ID: US11029845B2 - Status: Granted - Kind: B2 - Application number: 16/509,267 - Filed: 2019-07-11 - Published: 2021-01-14 - Granted: 2021-06-08 - Earliest claimed: 2019-07-11 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Christopher M. Becker; Nazeeh A. Eldirghami; Kevin W. Barnes; Julia Schwarz; Eric Carter - Web page: https://patentgrep.com/patent/US11029845B2 ## CPC - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The discussion relates to virtual keyboard engagement. One example can define key volumes relating to keys of a virtual keyboard and detect finger movement of a user through individual key volumes. The example can detect parameter changes associated with detected finger movement through individual key volumes and build potential key sequences from detected parameter changes. ## Claims ### Claim 1 A device, comprising: a display component configured to be positioned on a head of a user in front of the user's eyes and configured to generate virtual content that appears to the user in a scene and that includes virtual keys; sensors configured to monitor frustums defined by individual virtual keys that extend from the user's eyes through the individual virtual keys relative to the user; and, a processor configured to assign first probabilities to user movement through individual frustums that is parallel to the virtual content and to assign second higher probabilities to user movement through the individual frustums that is toward or away from the user and to predict an intended user action relative to the virtual content based at least in part on the first probabilities and the second probabilities. ### Claim 2 (depends on 1) The device of claim 1, wherein the sensors are configured to detect parameters associated with the movement of a user extremity through the frustums. ### Claim 3 (depends on 2) The device of claim 2, wherein the parameters relate to velocity and direction relative to the individual frustums. ### Claim 4 (depends on 3) The device of claim 3, wherein the virtual content is associated with the individual virtual keys associated with characters, and wherein the processor is configured to compare a sequence of characters associated with the movement to a dictionary of known words to predict the intended user action as spelling an individual word from the dictionary. ### Claim 5 (depends on 4) The device of claim 4, wherein the processor is configured to perform the intended user action. ### Claim 6 (depends on 5) The device of claim 5, wherein the processor is configured to perform the intended user action by auto-populating the individual word into a dialog box or providing the individual word as an auto-suggestion for the dialog box. ### Claim 7 (depends on 6) The device of claim 6, wherein the processor is configured to perform the auto-populating when a confidence associated with the individual word exceeds a threshold. ### Claim 8 A device-implemented method, comprising: generating a virtual keyboard with head mounted optics positioned in front of eyes of a user; detecting a path of a user extremity in 3D space proximate to a perceived position of the virtual keyboard; assigning probabilities to individual keys of the virtual keyboard when the path passes through a frustum defined at least in part by the individual keys; updating the probabilities relative to an individual key to a higher probability when the path relative to the individual key includes movement either toward or away from the user in the frustum; predicting an intended user action based at least in part on the probabilities and the updated probabilities; and, automatically taking an action on behalf of the user based at least in part upon the predicted intended user action. ### Claim 9 (depends on 8) The device-implemented method of claim 8, wherein detecting a path of a user extremity comprises detecting the path of a finger, multiple fingers, a hand, fingers of both a left and right hand, and/or an object held in the hand. ### Claim 10 (depends on 8) The device-implemented method of claim 8, wherein assigning probabilities to individual keys of the virtual keyboard comprises assigning a first probability to the individual key of the keyboard when the path passes relative to the individual key under first parameters and assigning a second higher probability to another individual key when the path passes relative to the another individual key under second parameters. ### Claim 11 (depends on 10) The device-implemented method of claim 10, wherein taking an action comprises populating a dialog box with a word selected based at least in part upon the at least one of the first probability, the second higher probability, and/or the updated probabilities. ### Claim 12 A device-implemented method, comprising: generating virtual content with a device positioned on a head of a user, the virtual content including virtual keys and appearing to the user in a volume beyond the device; defining virtual content frustums extending from the device and around a periphery of individual virtual keys, detecting finger movement of a user through individual virtual content frustums; detecting parameter changes associated with detected finger movement through the individual virtual content frustums, the parameter changes distinguishing finger movement through the virtual content frustums that is generally parallel to the virtual content from finger movement that is toward or away from the user relative to individual frustums; and, building potential user selections of individual virtual keys from detected parameter changes at least in part by weighting the finger movement toward or away from the user higher than finger movement parallel to the virtual content. ### Claim 13 (depends on 12) The device-implemented method of claim 12, wherein generating virtual content comprises generating a virtual keyboard and the defining virtual content frustums are defined at least in part by eyes of the user and the periphery of the individual virtual keys. ### Claim 14 (depends on 13) The device-implemented method of claim 13, wherein detecting parameter changes comprises detecting at least one of changes in direction and changes in velocity. ### Claim 15 (depends on 14) The device-implemented method of claim 14, wherein detecting parameter changes in direction comprises at least one of: detecting changes in direction parallel to a key surface and detecting changes in a direction perpendicular to the key surface. ### Claim 16 (depends on 15) The device-implemented method of claim 15, wherein the detecting parameter changes further comprises assigning higher probabilities to individual virtual keys in potential key sequences associated with parameter changes. ### Claim 17 (depends on 16) The device-implemented method of claim 16, further comprising comparing the potential key sequences to known words. ### Claim 18 (depends on 16) The device-implemented method of claim 16, further comprising comparing the potential key sequences to known user commands.