# US 12,183,074 B2 · Dynamic input system for smart glasses based on user availability states > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,074 B2 - Document ID: US12183074B2 - Status: Granted - Kind: B2 - Application number: 18/229,401 - Filed: 2023-08-02 - Published: 2023-11-23 - Granted: 2024-12-31 - Earliest claimed: 2021-10-22 - Assignees: Kyndryl, Inc. - Applicants: Kyndryl, Inc. - Inventors: Cesar Augusto Rodriguez Bravo; David Alonso Campos Batista; Sarbajit K. Rakshit; Diane G. Flemming - Web page: https://patentgrep.com/patent/US12183074B2 ## CPC - G06V 20/20: in augmented reality scenes - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06N 20/00: Machine learning - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/107: {Static hand or arm} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A dynamic user interface (UI) input system and method for smart glasses based on availability states of a user is provided. In embodiments, a method includes determining, by a computing device of smart glasses, an initial availability state of a user's hands for a user interface (UI) navigation event based on incoming image data; selecting, by the computing device, a UI configuration and a control profile for the UI navigation event from stored UI configurations and control profiles of the user based on the initial availability state of the user's hands; and initiating, by the computing device, the UI navigation event based on the selected UI configuration and the control profile, wherein content is displayed in a virtual UI of the smart glasses according to the UI configuration, and UI navigation by the user is controlled based on the control profile. ## Claims ### Claim 1 A method, comprising: selecting, by a computing device, a user interface (UI) configuration and a control profile for a UI navigation event, wherein the UI configuration comprises rules regarding how content will be displayed on the UI and the control profile comprises rules regarding gesture-based computing and navigating; initiating, by the computing device, the UI navigation event based on the selected UI configuration and the control profile; obtaining, by the computing device, usage information of the user comprising statistics used by a scoring system to rank an importance of content to be shown on the UI and how the ranked content is to be shown to the user on the UI; analyzing, by the computing device, the usage information of the user utilizing a machine learning algorithm to determine patterns of user behavior; and storing, by the computing device, the patterns of user behavior. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining, by the computing device, a user context comprising a location of the user, wherein the determining UI configuration and the control profile are determined based on the determined user context. ### Claim 3 (depends on 1) The method of claim 1, wherein application icons are displayed in a virtual UI of smart glasses according to the selected UI configuration, and hand movements by the user are interpreted based on the selected control profile. ### Claim 4 (depends on 3) The method of claim 3, wherein the virtual UI of the smart glasses comprises an augmented reality UI. ### Claim 5 (depends on 1) The method of claim 1, further comprising obtaining, by the computing device, incoming image data from one or more cameras and determining a location of the user and an initial availability state of a user's hands for the UI navigation event based on incoming image data. ### Claim 6 (depends on 5) The method of claim 5, wherein the determining the initial availability state of the user's hands comprises analyzing the image data utilizing image recognition and object recognition. ### Claim 7 (depends on 5) The method of claim 5, further comprising: continuously monitoring, by the computing device, the incoming image data for a change in the initial availability state of the user; determining, by the computing device, a change from the initial availability state of the user to a different availability state of the user based on the monitoring; and initiating, by the computing device, a UI function based on the determining the change. ### Claim 8 (depends on 7) The method of claim 7, wherein the UI function is selected from at least one of the group consisting of: changing the content displayed on a virtual UI; changing the configuration of the content displayed on the virtual UI; and changing the control profile of the user, thereby changing control rules utilized by the computing device during the navigation event. ### Claim 9 (depends on 7) The method of claim 7, wherein the UI function changes a display of content by a virtual UI so that content is displayed on a side of a display coinciding with an available hand of the user. ### Claim 10 (depends on 1) The method of claim 1, further comprising iteratively generating customized UI configurations and control profiles of the user based on iteratively determined patterns of user behavior performed via the machine learning algorithm, storing the UI customized configurations and control profiles of the user, and iteratively updating the customized UI configurations and control profiles of the user. ### Claim 11 A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to cause a computing device to: select a UI configuration and a control profile for a UI navigation event from stored UI configurations and control profiles, wherein the UI configurations each comprise rules regarding how content will be displayed on a UI and the control profiles each comprise rules regarding gesture-based computing and navigating; initiate the UI navigation event based on the selected UI configuration and the control profile; obtain usage information of the user comprising statistics used by a scoring system to rank an importance of content to be shown on the UI and how the ranked content is to be shown to the user on the UI; analyze the usage information of the user utilizing a machine learning algorithm to determine patterns of user behavior; and store the patterns of user behavior. ### Claim 12 (depends on 11) The computer program product of claim 11, further comprising displaying application icons in a virtual UI according to the UI configuration, and hand movements by the user are interpreted based on the control profile. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein the program instructions are further executable to cause the computing device to determine a user context, wherein the determining the UI configuration and the control profile is further based on the determined user context. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein the program instructions are further executable to cause the computing device to obtain incoming image data from one or more cameras, and further comprising determining initial availability state of the user comprises analyzing the image data utilizing image recognition and object recognition comprising object classification, object detection, and image segmentation of the image data. ### Claim 15 (depends on 12) The computer program product of claim 12, wherein the program instructions are further executable to cause the computing device to: continuously monitor incoming image data for a change in initial availability state of the user and a change in location of the user; determine a change from the initial availability state of the user to a different availability state of the user based on the monitoring; and initiate a UI function based on the determining the change in the different availability state of the user and the change in location of the user. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the UI function changes a display of content by a virtual UI so that content is displayed on a side of a display of smart glasses coinciding with an available hand of the user. ### Claim 17 A system comprising: a processor, a computer readable memory, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to cause a computing device to: determine an initial availability state of a user for a user interface (UI) navigation event based on incoming image data; select a UI configuration and a control profile for the UI navigation event from stored UI configurations and control profiles based on the initial availability state of the user and a context of the user, the UI configurations each comprising rules regarding how content will be displayed on the UI and the control profiles each comprising rules regarding gesture-based computing and navigating; and initiate the UI navigation event based on the selected UI configuration and the control profile; obtain usage information of the user comprising statistics used by a scoring system to rank an importance of content to be shown on the UI and how the ranked content is to be shown to the user on the UI; analyze the usage information of the user utilizing a machine learning algorithm to determine patterns of user behavior; and store the patterns of user behavior. ### Claim 18 (depends on 17) The system of claim 17, further comprising displaying application icons in a virtual UI according to the UI configuration. ### Claim 19 (depends on 17) The system of claim 17, wherein the program instructions are further executable to cause the computing device to: continuously monitor incoming image data to determine a change in an initial availability state of the user; determine a change from the initial availability state of the user to a different availability state of the user based on the monitoring; and initiate a UI function based on the determining the change.