# US 10,768,780 B2 · Method and system for context-driven displaying of shortcuts on touchscreen > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,780 B2 - Document ID: US10768780B2 - Status: Granted - Kind: B2 - Application number: 15/836,932 - Filed: 2017-12-11 - Published: 2019-06-13 - Granted: 2020-09-08 - Earliest claimed: 2017-12-11 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Ashley D Gritzman; Abdigani M Diriye; Komminist Weldemariam - Web page: https://patentgrep.com/patent/US10768780B2 ## CPC - G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34) - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A computer-implemented method of controlling a computer system is provided. In particular, the method relates to controlling of a touchscreen of the computer system in order to display contextually relevant user interface objects on a graphical display of the touchscreen. The computer system includes a processor and the touchscreen includes a touch sensor as an input device. The method includes determining, by way of a user context determination module, a user context of at least one user of the computer system, selecting, using an icon selection module, one or more user interface objects which are contextually relevant to the determined context of the user, and displaying, using the graphical display of the touchscreen, the one or more user interface objects selected by the icon selection module. Determining the user's context may include monitoring the user's behavior, mood or cognitive state. ## Claims ### Claim 1 A computer-implemented method of controlling a computer system which includes a processor and a touchscreen, the touchscreen including a graphical display and a touch sensor, the method including: determining, by way of a user context determination module, a user context of at least one user of the computer system; predicting, based on the user context, a probability of the user interacting with one or more user interface objects in relation to an application running on the computer system, the probability changing dynamically at least based on frequency of use of the one or more user interface objects; based on the predicting, selecting, using an icon selection module, one or more user interface objects which are contextually relevant to the determined context of the user; and displaying, using the graphical display of the touchscreen, the one or more user interface objects selected by the icon selection module, wherein the one or more user interface objects, which are displayed, change dynamically based on the application running on the computer system and the user context, wherein a position and size of the one or more user interface objects change dynamically based on the probability. ### Claim 2 (depends on 1) A method as claimed in claim 1, wherein determining the user context includes monitoring behavior of the user by way of at least one input device of the computer system. ### Claim 3 (depends on 2) A method as claimed in claim 2, which includes capturing, using a video camera coupled to the processor of the computer system, video footage of the user and establishing a mood or cognitive state of the user by way of an image processing module based on the video footage. ### Claim 4 (depends on 2) A method as claimed in claim 2, which includes capturing, using a sound recorder coupled to the processor, audio footage of the user and establishing a mood or cognitive state of the user by way of an audio processing module based on the audio footage. ### Claim 5 (depends on 2) A method as claimed in claim 2, which includes sensing, using the touch sensor of the touchscreen, pressure applied to the touchscreen by the user in order to infer a mood or cognitive state of the user, and selecting, using the icon selection module, one or more user interface objects to display on the graphical display of the touchscreen based upon the mood or cognitive state of the user. ### Claim 6 (depends on 2) A method as claimed in claim 2, wherein monitoring behavior of the user includes: capturing user interaction with the computer system using any one or more computer input devices selected from: the touch sensor, a keyboard, a pointing device, video camera and sound recorder. ### Claim 7 (depends on 6) A method as claimed in claim 6, which includes: evaluating, using the user context determination module, the captured user interaction by identifying a sequence of interactions the user has with the computer system; developing a user interface object prediction model based upon the sequence of interactions; and predictively displaying, on the graphical display of the touchscreen, user interface objects based upon the user interface object prediction model. ### Claim 8 (depends on 1) A method as claimed in claim 1, which includes associating at least one user interface object with an age of the user and wherein determining the user context includes determining an age of the user. ### Claim 9 (depends on 1) A method as claimed in claim 1, which includes keeping record, using the user context determination module, of the user's interaction with respective user interface objects and ranking, using a user interface object prediction model, the user interface objects based upon their frequency of use by the user. ### Claim 10 (depends on 6) A method as claimed in claim 6, which includes associating, using the user interface object prediction model, a group of user interface objects with a particular user based upon that user's interaction with the group of user interface objects. ### Claim 11 (depends on 1) A method as claimed in claim 1, wherein determining the user context includes establishing, using a location determination module, a location of the computer system and selecting, using the icon selection module, one or more user interface objects to display based upon the established location of the computer system. ### Claim 12 (depends on 1) A method as claimed in claim 1, wherein determining the user context includes establishing, using a time keeping module, a time of day and selecting, using the icon selection module, one or more user interface objects to display based upon the time of day. ### Claim 13 (depends on 1) A method as claimed in claim 1, wherein determining the user context includes sensing ambient light intensity and displaying, on the graphical display of the touchscreen, a user interface object which can be manipulated in order to adjust a screen brightness. ### Claim 14 (depends on 1) A method as claimed in claim 1, wherein determining the user context includes monitoring, using a calendar module of the computer system, a schedule of the user and displaying a contextually relevant user interface object on the graphical display of the touchscreen related to a scheduled calendar event. ### Claim 15 (depends on 1) A method as claimed in claim 1, wherein selecting one or more user interface objects includes extracting, using the icon selection module, one or more user interface objects from a graphical user interface represented on a primary display of the computer system and displaying the extracted user interface objects on the graphical display of the touchscreen. ### Claim 16 (depends on 1) A method as claimed in claim 1, wherein selecting includes defining user specified rules for display of user interface objects on the graphical display of the touchscreen. ### Claim 17 A computer system for controlling a touchscreen which includes a graphical display and a touch sensor, the computer system including: a processor; the touchscreen, coupled to the processor; and a computer readable storage medium having stored thereon program instructions executable by the processor to direct the computer system to: determine a user context by way of a user context determination module; predict, based on the user context, which one or more user interface objects the user is likely to interact a probability of the user interacting with one or more user interface objects in relation to an application currently running on the computer system, the probability changing dynamically at least based on frequency of use of the one or more user interface objects; based on the predicting, select, using an icon selection module, one or more user interface objects which are contextually relevant to the context of the user; and display the one or more user interface objects selected by the icon selection module using the graphical display of the touchscreen, wherein the one or more user interface objects, which are displayed, change dynamically based on the application running on the computer system and the user context, wherein a position and size of the one or more user interface objects change dynamically based on the probability. ### Claim 18 (depends on 17) A computer system as claimed in claim 17, wherein the program instructions executable by the processor direct the computer system to monitor behavior of the user by way of at least one input device of the computer system. ### Claim 19 (depends on 18) A computer system as claimed in claim 18, wherein the input device is a video camera which is communicatively coupled to the processor. ### Claim 20 A computer program product for controlling a computer system which includes a processor and a touchscreen which includes a graphical display and a touch sensor, the computer program product comprising a computer-readable medium having program instructions stored thereon which are executable by the processor of the computer system to enable the processor to: determine a user context by way of a user context determination module; predict, based on the user context, a probability of the user interacting with one or more user interface objects in relation to an application running on the computer system, the probability changing dynamically at least based on frequency of use of the one or more user interface objects; based on the predicting, select, using an icon selection module, one or more user interface objects which are contextually relevant to the user context; and display the one or more selected user interface objects on the graphical display of the touchscreen, wherein the one or more user interface objects, which are displayed, change dynamically based on the application running on the computer system and the user context, wherein a position and size of the one or more user interface objects change dynamically based on the probability.