# US 10,872,199 B2 · Mapping a gesture and/or electronic pen attribute(s) to an advanced productivity action > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,199 B2 - Document ID: US10872199B2 - Status: Granted - Kind: B2 - Application number: 15/990,598 - Filed: 2018-05-26 - Published: 2019-11-28 - Granted: 2020-12-22 - Earliest claimed: 2018-05-26 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Elise Leigh Livingston; Daniel Yancy Parish; Adam Samuel Riddle - Web page: https://patentgrep.com/patent/US10872199B2 ## CPC - G06F 40/171: by use of digital ink - G06F 3/03545: {Pens or stylus} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Described herein is a system and method for modifying electronic documents. While a user is editing an electronic document on a canvas of an application, a trigger event related to an electronic pen is received (e.g., explicitly or inferred). The electronic pen has one or more associated attributes (e.g., type of pen, color of pen, thickness of line, transparency value). In response to the trigger event, which of a plurality of advanced productivity actions related to editing to apply to the electronic document is determined based upon at least one of the associated attributes. The advanced production actions can include, for example, styles, formatting, and/or themes. The electronic document is modified in accordance with the determined advanced productivity action. ## Claims ### Claim 1 A system comprising: a processor and a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the system to: provide mappings of different pen marking types to different table actions on tables of electronic documents, the mappings including a first mapping of a first pen marking type to a predefined table style for formatting tables of electronic documents and a second mapping of a second pen marking type to an advanced table manipulation that involves adding or removing cells from the tables of the electronic documents; while a user is editing a particular table of a particular electronic document on a canvas of an application, receive a trigger event that changes an electronic pen to a selected pen marking type; based at least on the selected pen marking type and the mappings, determine a particular table action to apply to the particular table of the particular electronic document, wherein the particular table action comprises applying the predefined table style to the particular table in first instances when the selected pen marking type is the first pen marking type and the particular table action comprises the advanced table manipulation in second instances when the selected pen marking type is the second pen marking type; and modify the particular table of the particular electronic document in accordance with the particular table action. ### Claim 2 (depends on 1) The system of claim 1, wherein the computer-executable instructions, when executed by the processor, cause the system to: receive a gesture comprising a digital ink stroke; and modify the particular table responsive to the received gesture. ### Claim 3 (depends on 2) The system of claim 2, wherein the computer-executable instructions, when executed by the processor, cause the system to: determine the particular table action to apply based on both the selected pen marking type and content of a portion of the particular electronic document that is in proximity to the received gesture. ### Claim 4 (depends on 1) The system of claim 1, wherein the selected pen marking type relates to at least one of a color of markings made by the electronic pen, a thickness of line markings made by the electronic pen, or a transparency value of markings made by the electronic pen. ### Claim 5 (depends on 1) The system of claim 1, wherein the computer-executable instructions, when executed by the processor, cause the system to: determine the particular table action to apply based on both the selected pen marking type and a semantic structure of a portion of the particular electronic document that is in proximity to a received gesture. ### Claim 6 (depends on 1) The system of claim 1, wherein the advanced table manipulation involves adding a row to the particular table in the second instances when the selected pen marking type is the second pen marking type. ### Claim 7 (depends on 1) The system of claim 1, wherein the advanced table manipulation involves adding a column to the particular table in the second instances when the selected pen marking type is the second pen marking type. ### Claim 8 (depends on 1) The system of claim 1, wherein the advanced table manipulation involves merging cells of the particular table in the second instances when the selected pen marking type is the second pen marking type. ### Claim 9 (depends on 1) The system of claim 1, wherein the advanced table manipulation involves splitting cells of the particular table in the second instances when the selected pen marking type is the second pen marking type. ### Claim 10 (depends on 1) The system of claim 1, wherein the trigger event is received based at least upon an explicit action of a user. ### Claim 11 (depends on 1) The system of claim 1, wherein the trigger event is inferred based at least upon a user's previous history with the system. ### Claim 12 (depends on 1) The system of claim 1, wherein the computer-executable instructions, when executed by the processor, cause the system to: determine which of the different table actions to apply based at least upon respective probabilities for the different table actions, the probabilities being determined using the selected pen marking type, content of the particular table, and a semantic structure of the particular table. ### Claim 13 A method, comprising: mapping a first type of pen marking to a predefined table style for formatting tables of electronic documents; mapping a second type of pen marking to an advanced table manipulation that involves adding or removing cells from the tables of the electronic documents; receiving a trigger event related to an electronic pen, the trigger event indicating a change to one or more marking attributes of the electronic pen; in response to the trigger event, selecting a particular action to apply to a particular table of a particular electronic document based at least upon the one or more marking attributes of the electronic pen, wherein the particular action comprises applying the predefined table style to the particular table in instances when the trigger event indicates a change of the one or more marking attributes to the first type of pen marking and applying the advanced table manipulation to the particular table in other instances when the trigger event indicates a change of the one or more marking attributes to the second type of pen marking; and modifying the particular table of the particular electronic document in accordance with the particular action. ### Claim 14 (depends on 13) The method of claim 13, wherein the first type of pen marking comprises highlighter markings and the second type of pen marking comprises regular pen markings. ### Claim 15 (depends on 14) The method of claim 14, wherein the modifying comprises: performing the advanced table manipulation on the particular table in the other instances when the trigger event changes from the highlighter markings to the regular pen markings; and applying the predefined table style to the particular table in the instances when the trigger event changes from the regular pen markings to the highlighter markings. ### Claim 16 (depends on 15) The method of claim 15, wherein the advanced table manipulation performed on the particular table in the other instances when the trigger event changes from the highlighter markings to the regular pen markings includes one or more of adding rows to the particular table, adding columns to the particular table, merging cells in the particular table, or splitting cells in the particular table. ### Claim 17 (depends on 13) The method of claim 13, wherein the advanced table manipulation comprises one or more of adding a row to the particular table, adding a column to the particular table, merging cells of the particular table, or splitting cells of the particular table. ### Claim 18 A computer storage media storing computer-readable instructions that, when executed, cause a computing device to: during editing of a particular table of a particular electronic document on a canvas of an application, receive a trigger event related to an electronic pen, the trigger event selecting a particular type of pen marking for the electronic pen; access mappings of different pen marking types to different table actions, the mappings including a first mapping of a first pen marking type to a predefined table style for formatting tables of electronic documents and a second mapping of a second pen marking type to an advanced table manipulation that involves adding or removing cells from the tables of the electronic documents; based at least upon the mappings and the particular type of pen marking selected for the electronic pen, determine a particular table action to apply to the particular table of the particular electronic document, wherein the particular table action comprises applying the predefined table style to the particular table in first instances when the particular type of pen marking is the first pen marking type and the particular table action comprises the advanced table manipulation in second instances when the particular type of pen marking is the second pen marking type; and modify the particular table of the particular electronic document in accordance with the particular table action. ### Claim 19 (depends on 18) The computer storage media of claim 18, wherein the computer-readable instructions, when executed, cause the computing device to: receive a gesture comprising a digital ink stroke; and perform the particular table action in response to the gesture. ### Claim 20 (depends on 18) The computer storage media of claim 18, wherein the first pen marking type mapped to the predefined table style comprises a highlighter marking, and the second pen marking type mapped to the advanced table manipulation comprises a regular pen marking.