# US 2017/0285755 A1 · Methods for analysing user interactions with a user interface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0285755 A1 - Document ID: US20170285755A1 - Status: Published application - Kind: A1 - Application number: 15/083,640 - Filed: 2016-03-29 - Published: 2017-10-05 - Earliest claimed: 2016-03-29 - Assignees: SessionCam Limited - Applicants: SessionCam Limited - Inventors: Richard Churchill; Joao Duarte - Web page: https://patentgrep.com/patent/US20170285755A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06N 5/04: Inference or reasoning models - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Computer-implemented methods for analysing interactions between a user and a computing device include analysing user interaction data to identify a plurality of user gestures, and analysing the gestures to identify those likely to represent user struggles or difficulties encountered with a user interface. A predictive model is generated from this analysis for use in identifying user gestures likely to represent user struggles during further user sessions. ## Claims ### Claim 1 A computer-implemented method of analysing interactions between a user and a computing device, the method using at least one data processor and electronic data storage and comprising the steps of: (a) storing user interaction data in the data storage, wherein the user interaction data is derived from user interactions with the computing device using an electronic input device during at least one user session, and with the at least one data processor: (b) retrieving the user interaction data from the data storage; (c) analysing the user interaction data to identify a plurality of user gestures; (d) analysing the plurality of user gestures to identify user gestures likely to represent user struggles; (e) generating a predictive model with reference to the analysis of step (d) for use in analysing user gestures to identify user gestures likely to represent user struggles; and (f) outputting the predictive model to the data storage. ### Claim 2 (depends on 1) The method of claim 1, wherein step (d) comprises analysing each user gesture to determine a numerical value for each of a plurality of gesture properties in relation to each user gesture. ### Claim 3 (depends on 2) The method of claim 2, wherein the gesture properties correspond to at least one of: a duration of the gesture, coverage of a gesture, an average speed of the gesture, a maximum speed, an average acceleration during the gesture, a maximum acceleration, a distance travelled during the gesture, a rotation defined with reference to the gesture, and a direction of the gesture. ### Claim 4 (depends on 2) The method of claim 2, wherein step (d) includes determining and applying a scaling process to the numerical values for at least one of the gesture properties. ### Claim 5 (depends on 2) The method of claim 2, wherein step (d) includes defining a plurality of gesture clusters, with each cluster comprising a plurality of the user gestures, wherein each cluster is defined with reference to at least one of the gesture properties. ### Claim 6 (depends on 5) The method of claim 5, wherein step (d) includes assigning a weight value to each gesture cluster, with the weight value indicating a relative importance of the cluster in relation to the other clusters. ### Claim 7 (depends on 6) The method of claim 6, wherein the weight value is determined using at least one of the following techniques: TF/IDF, sublinear TF scaling and maximum TF normalisation. ### Claim 8 (depends on 5) The method of claim 5, wherein step (d) includes assigning a struggle value to each gesture cluster, with the struggle value indicating the likelihood that the user gestures of the gesture cluster represent user struggles. ### Claim 9 (depends on 1) The method of claim 1, wherein step (d) includes applying a dimensionality reduction method to a plurality of gesture properties. ### Claim 10 (depends on 1) The method of claim 1, wherein the input device is selected from: a mouse, a touch-sensitive screen, a joystick, and a system able to detect remote user movements. ### Claim 11 (depends on 1) The method of claim 1, wherein in step (c), a start and end of each user gesture is defined by a period of inactivity of the input device longer than a predetermined threshold. ### Claim 12 (depends on 1) The method of claim 1, wherein each user gesture identified in step (c) comprises a number of user interaction data points which is greater than a predetermined threshold. ### Claim 13 (depends on 1) The method of claim 1, wherein step (c) comprises arranging groups of consecutive user interactions into ngrams, each of which represents a user gesture. ### Claim 14 (depends on 1) A non-transitory recording medium storing computer interpretable instructions for causing at least one data processor to perform the method in accordance with claim 1. ### Claim 15 (depends on 1) A computing device comprising a memory device including computer interpretable instructions that, when executed, direct the computing device to perform the method of claim 1, and at least one data processor. ### Claim 16 (depends on 1) A computer-implemented method of analysing user interactions with a computing device to identify user interactions likely to represent user struggles, wherein the method uses at least one data processor and electronic data storage storing the predictive model generated using the method of claim 1, and the method comprises the steps of: (i) storing user interaction data in the data storage, wherein the user interaction data is derived from user interactions with the computing device using an electronic input device during a user session, and with the at least one data processor: (ii) retrieving the user interaction data from the data storage; (iii) analysing the user interaction data to identify a plurality of user gestures; (iv) retrieving the predictive model from the data storage; and (v) analysing the plurality of user gestures using the predictive model to identify the user gestures likely to represent user struggles. ### Claim 17 (depends on 16) The method of claim 16, wherein the user interaction data is derived from user interactions during a plurality of user sessions involving a computer-implemented graphical interface, and the method includes the step of identifying a user session or a portion of the interface which is associated with user gestures likely to represent user struggles. ### Claim 18 (depends on 17) The method of claim 17, wherein a plurality of sessions or portions of the interface are identified which are associated with user gestures likely to represent user struggles. ### Claim 19 (depends on 18) The method of claim 18, wherein the at least one data processor calculates a respective struggle score for each of the sessions or portions indicative of the likelihood that it is associated with user gestures that represent user struggles. ### Claim 20 (depends on 19) The method of claim 19, wherein the at least one data processor determines a priority order for the sessions or portions with reference to their respective struggle scores. ### Claim 21 (depends on 17) The method of claim 17, wherein the at least one data processor calculates a count value for a selected session or a portion of the interface with reference to how many of the user sessions involved user gestures likely to represent user struggles with the selected session or portion of the interface at a specified point in time or during a specified period of time. ### Claim 22 (depends on 21) The method of claim 21, wherein the at least one data processor outputs signals for sending to a display device so as to cause the display device to display a plot of a parameter dependent on the count value for the selected session or portion of the interface against time. ### Claim 23 (depends on 19) The method of claim 19, wherein the at least one data processor calculates the count value for a plurality of selected sessions or portions of the interface, and determines a priority order for the selected sessions or portions with reference to their respective count values. ### Claim 24 (depends on 17) The method of claim 17, wherein the at least one data processor receives an input signal identifying a user session, and outputs signals for sending to a display device so as to cause the display device to display a replay of at least part of the user session. ### Claim 25 (depends on 16) A non-transitory recording medium storing computer interpretable instructions for causing at least one data processor to perform the method in accordance with claim 16. ### Claim 26 (depends on 16) A computing device comprising a memory device including computer interpretable instructions that, when executed, direct the computing device to perform the method of claim 16, and at least one data processor.