# US 2019/0318652 A1 · Use of intelligent scaffolding to teach gesture-based ink interactions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318652 A1 - Document ID: US20190318652A1 - Status: Published application - Kind: A1 - Application number: 15/953,101 - Filed: 2018-04-13 - Published: 2019-10-17 - Earliest claimed: 2018-04-13 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Elise LIVINGSTON; Adam Samuel RIDDLE; Allison SMEDLEY; Robin TROY - Web page: https://patentgrep.com/patent/US20190318652A1 ## CPC - G09B 19/0053: {Computers, e.g. programming} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06N 20/00: Machine learning - G06V 30/347: {Sampling; Contour coding; Stroke extraction} - G06V 30/36: {Matching; Classification} - G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various embodiments of the present technology relate to digital inking technology. More specifically, some embodiments relate to use of intelligent scaffolding to teach gesture-based ink interactions. For example, in some embodiments telemetry data on user interactions with a user interface to an application can be collected at a client device. The telemetry data can be analyzed to identify user proficiency with a digital inking gesture. Upon determining a low user proficiency with the digital inking gesture, user interactions resembling a digital inking gesture within the application can be identified. A training interface can be automatically surfaced, on a display of the client device, with specifically scoped training information on the digital inking gesture to improve the user proficiency with the digital inking gesture. ## Claims ### Claim 1 A computing apparatus comprising: one or more computer readable storage media; a processing system operatively coupled with the one or more computer readable storage media; and program instructions stored on the one or more computer readable storage media that, when executed by the processing system, direct the processing system to at least: collect data on user interactions within an application; analyze data on the user interactions to identify user proficiency with digital inking gestures within the application; and render a user interface with training information to improve the user proficiency with digital inking gestures. ### Claim 2 (depends on 1) The computing apparatus of claim 1, wherein the user interactions with the user interface collected in the data includes repeated inking gestures followed by undo requests and wherein to analyze the data, the program instructions direct the processing system to: identify the repeated inking gestures followed by the undo requests; analyze the repeated inking gestures to identify actual inking gestures supported by the application that are similar to the repeated inking gestures; and access the training information associated with the actual inking gestures to be rendered on the user interface. ### Claim 3 (depends on 1) The computing apparatus of claim 1, wherein the program instructions further direct the processing system to recognize failed digital inking gestures and analyze the failed digital inking gestures to identify, using a machine learning classifier, actual inking gestures resembling a digital inking feature within the application. ### Claim 4 (depends on 1) The computing apparatus of claim 1, wherein the program instructions further direct the processing system to render the user interface with the training information when the user interface has not been presented before, after a time period has elapsed from previous presentations of the user interface, or upon new digital inking features becoming available within the application. ### Claim 5 (depends on 1) The computing apparatus of claim 1, wherein the user interactions are collected upon detection of the application entering a ink editing mode allowing ink gestures to be translated into editing commands and wherein the data includes keyboard interactions, mouse interactions, inking gestures, and digital pen interactions with the user interface to the application. ### Claim 6 (depends on 1) The computing apparatus of claim 1, wherein the program instructions direct the processing system to determine whether the user interactions include a first use of digital ink within the application and surface the user interface with general digital ink training information. ### Claim 7 (depends on 1) The computing apparatus of claim 1, wherein the program instructions direct the processing system to record interactions with the user interface and transmit the data and interactions with the user interface to a cloud-based data repository to be ingested by a machine learning system, along with other data and interactions with additional user interfaces, to determine rules regarding when to render the user interface. ### Claim 8 A method comprising: collecting, at a client device, telemetry data on user interactions with a user interface to an application; analyzing telemetry data on the user interactions to identify user proficiency with a digital inking gesture; identifying, upon determining a low user proficiency with the digital inking gesture, user interactions resembling a digital inking gesture within the application; and automatically surfacing, on a display of the client device, a user interface with training information on the digital inking gesture to improve the user proficiency with digital inking gesture. ### Claim 9 (depends on 8) The method of claim 8, wherein the user interactions with the user interface collected in the telemetry data include repeated inking gestures followed by undo requests and wherein to analyzing the telemetry data to identify user proficiency includes: identifying the repeated inking gestures followed by the undo requests; and analyzing the repeated inking gestures to identify the digital inking gesture supported by the application that are similar to the repeated inking gestures. ### Claim 10 (depends on 9) The method of claim 9, wherein analyzing the repeated inking gestures to identify the digital inking gesture within the application using a machine learning classifier. ### Claim 11 (depends on 9) The method of claim 9, further comprising retrieving the training information associated with the digital inking gesture to be rendered on the user interface; and wherein the user interface is automatically surfaced only if the user interface has not been presented before, after a time period has elapsed from a previous presentation of the user interface, or upon new digital inking features becoming available within the application. ### Claim 12 (depends on 8) The method of claim 8, wherein the user interactions collected by the telemetry data include keyboard interactions, mouse interactions, inking gestures, and digital pen interactions with the user interface to the application. ### Claim 13 (depends on 8) The method of claim 8, further comprising analyzing the telemetry data to determine whether the user interactions include a first use of digital ink within the application, and upon determining a first use, automatically surfacing the user interface with general digital ink training information. ### Claim 14 (depends on 8) The method of claim 8, further comprising: recording interactions with the user interface; and transmitting the telemetry data and interactions with the user interface to a cloud-based data repository to be ingested by a machine learning system, along with other telemetry data and interactions with additional user interfaces, to determine rules regarding when to automatically surface the user interface. ### Claim 15 (depends on 14) The method of claim 14, further comprising: receiving, at the client device, the rules regarding when to automatically surface the user interface; and updating the application with the rules. ### Claim 16 One or more computer readable storage media having program instructions stored thereon for supporting digital inking training that, when executed by one or more processor, direct a machine to at least: monitor user interactions with a user interface to an application; analyze the user interactions to identify user proficiency with digital inking gestures or to identify user interactions resembling a digital inking feature within the application; and automatically render a user interface with training information to improve the user proficiency with digital inking gestures or to train on use of the digital inking feature. ### Claim 17 (depends on 16) The one or more computer readable storage media of claim 16, wherein to analyze the user interactions to identify a digital inking feature, the machine uses a machine learning classifier. ### Claim 18 (depends on 16) The one or more computer readable storage media of claim 16, wherein the program instructions further cause the machine to record interactions with the user interface and transmit the user interactions and interactions with the user interface to a cloud-based analysis platform to be ingested by a machine learning system to determine rules regarding when to render the user interface. ### Claim 19 (depends on 17) The one or more computer readable storage media of claim 17, wherein the program instructions further cause the machine to determine whether the user interactions include a first use of digital ink within the application and automatically render the user interface with digital ink training information that includes most commonly used gestures. ### Claim 20 (depends on 17) The one or more computer readable storage media of claim 17, wherein the program instructions further cause the machine to suppress automatically rendering the user interface when a time period has not elapsed from a previous automatic rendering of the user interface.