# US 2021/0278965 A1 · System and method of guiding handwriting diagram input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278965 A1
- Document ID: US20210278965A1
- Status: Published application
- Kind: A1
- Application number: 17/205,246
- Filed: 2021-03-18
- Published: 2021-09-09
- Earliest claimed: 2015-10-19
- Assignees: MyScript
- Applicants: MyScript
- Inventors: Robin MéLINAND; Romain BEDNAROWICZ; Claire SIDOLI; Fabien RIC; Nicolas RUCINE; Erwan JESTIN; Anthony LAURENCE; Khaoula ELAGOUNI; Cyril CEROVIC; Anne BONNAUD
- Web page: https://patentgrep.com/patent/US20210278965A1

## CPC

- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/171: by use of digital ink
- G06V 30/32: Digital ink
- G06V 30/387: {using human interaction, e.g. selection of the best displayed recognition candidate}
- G06V 30/10: Character recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system, method and computer program product for guiding hand-drawing of diagrams including text and non-text elements on a computing device are provided. The computing device has a processor and a non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor. Display is performed, on an interactive display of the computing device, of a guide element associated with at least one diagram element of displayed handwriting diagram input. The guide element is configured with a depiction of the at least one diagram element in recognized form.

## Claims

### Claim 1

A system for guiding hand-drawing of diagrams including text and non-text element types on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the at least one non-transitory computer readable medium configured to:
accept input of, on an input area of an interactive display of the computing device:
a hand-drawing diagram element, and
a selection gesture;
cause classification of the hand-drawing diagram element as a first type element;
cause recognition of the first type hand-drawing diagram element based on the classification;
cause display of, on the interactive display:
a digital ink of the hand-drawing diagram element,
a dynamic prompter which dynamically depicts at least one recognized form of the hand-drawing diagram element,
wherein the computer readable medium is configured to cause display of the dynamic prompter in response to the selection gesture selecting the digital ink of the hand-drawing diagram element, and
a manual classification action;
cause selection of the manual classification action in response to receiving interaction with the dynamic prompter;
cause re-classification of the hand-drawing diagram element as being a second type element, in response to said selection; and
cause re-recognition of the hand-drawing diagram element in response to said re-classification;
wherein the dynamic prompter is configured to cause depiction of a recognized form of the re-recognized second type hand-drawing diagram element.

### Claim 2 (depends on 1)

A system as claimed in claim 1, wherein the dynamic prompter dynamically depicts the at least one recognized form of the hand-drawing diagram element as handwriting is input.

### Claim 3 (depends on 1)

A system as claimed in claim 1, wherein the first type element is non-text and the dynamic prompter depicts the at least one recognized form of the hand-drawing non-text diagram element in icon form.

### Claim 4 (depends on 3)

A system as claimed in claim 3, wherein the hand-drawing diagram element is a shape element of the diagram, the icon depicting the shape.

### Claim 5 (depends on 1)

A system as claimed in claim 1, wherein the first type element is text and the dynamic prompter depicts the at least one recognized form of the hand-drawing text diagram element in typeset ink.

### Claim 6 (depends on 1)

A system as claimed in claim 1, wherein
the hand-drawing diagram element is classified as non-text;
the selection of the manual classification action causes re-classification of the non-text element as text;
the re-classification causes re-recognition of the re-classified text element; and
the dynamic prompter causes depiction of a recognized form of the re-recognized text element.

### Claim 7 (depends on 1)

A system as claimed in claim 1, wherein
the hand-drawing diagram element is classified as text;
the selection of the manual classification action causes re-classification of the text element as non-text;
the re-classification causes re-recognition of the re-classified non-text element; and
the dynamic prompter causes depiction of a recognized form of the re-recognized non-text element.

### Claim 8

A method for guiding hand-drawing of diagrams including text and non-text element types on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:
inputting, on an input area of an interactive display of the computing device:
a hand-drawing diagram element, and
a selection gesture;
classifying the hand-drawing diagram element as a first type element;
recognizing the first type hand-drawing diagram element based on the classification;
displaying, on the interactive display:
a digital ink of the hand-drawing diagram element,
a dynamic prompter which dynamically depicts at least one recognized form of the hand-drawing diagram element,
wherein the dynamic prompter is displayed in response to selecting, via the selection gesture, the digital ink of the hand-drawing diagram element; and
a manual classification action;
selecting the manual classification action in response to receiving interaction with the dynamic prompter;
re-classifying the hand-drawing diagram element as being a second type element, in response to said selection; and
re-recognizing of the hand-drawing diagram element in response to re-classification;
wherein the dynamic prompter is configured to cause depiction of a recognized form of the re-recognized hand-drawing diagram element.

### Claim 9 (depends on 8)

A method as claimed in claim 8, wherein the dynamic prompter dynamically depicts the at least one recognized form of the hand-drawing diagram element as the handwriting is input.

### Claim 10 (depends on 8)

A method as claimed in claim 8, comprising displaying a recognized form of the hand-drawing diagram element overlaid on the digital ink of the diagram element wherein the first type element is non-text.

### Claim 11 (depends on 8)

A method as claimed in claim 8, comprising depicting the at least one recognized form of the hand-drawing diagram element in icon form wherein the first type element is non-text.

### Claim 12 (depends on 11)

A method as claimed in claim 11, wherein the hand-drawing diagram element is a shape element of the diagram, the icon depicting the shape.

### Claim 13 (depends on 8)

A method as claimed in claim 8, comprising depicting of the at least one recognized form of the hand-drawing diagram element in typeset ink wherein the first type element is text.

### Claim 14 (depends on 8)

A method as claimed in claim 8, comprising
classifying the hand-drawing diagram element as non-text;
re-classifying the non-text element as text following the selection of the manual classification action;
re-recognizing the re-classified text element; and
depicting a recognized form of the re-recognized text element by the dynamic prompter.

### Claim 15 (depends on 8)

A method as claimed in claim 8, comprising
classifying the hand-drawing diagram element as text;
re-classifying the text element as non-text following the selection of the manual classification action;
re-recognizing the re-classified non-text element; and
depicting a recognized form of the re-recognized non-text element by the dynamic prompter causes.

### Claim 16

A non-transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for guiding hand-drawing of diagrams including text and non-text element types on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:
inputting, on an input area of an interactive display of the computing device:
a hand-drawing diagram element, and
a selection gesture;
classifying the hand-drawing diagram element as either text or non-text;
recognizing the hand-drawing diagram element based on the classification;
displaying, on the interactive display:
a digital ink of the hand-drawing diagram element,
a dynamic prompter which dynamically depicts at least one recognized form of the hand-drawing diagram element,
wherein the dynamic prompter is displayed in response to selecting, via the selection gesture, the digital ink of the hand-drawing diagram element; and
a manual classification action;
selecting the manual classification action in response to receiving interaction with the dynamic prompter;
re-classifying of a text element or a non-text element as being non-text or text element respectively, in response to said selection; and
re-recognizing of the hand-drawing diagram element being performed based on the re-classification, in response to re-classification;
wherein the dynamic prompter is configured to cause depiction of a recognized form of the re-recognized hand-drawing diagram element.

### Claim 17 (depends on 16)

A non-transitory computer readable medium as claimed in claim 16, wherein the dynamic prompter dynamically depicts the diagram element as the handwriting is input.

### Claim 18 (depends on 16)

A non-transitory computer readable medium as claimed in claim 16, wherein the first type element is non-text and depiction of the at least one recognized form of the hand-drawing diagram element is in icon form.

### Claim 19 (depends on 18)

A non-transitory computer readable medium as claimed in claim 18, wherein the hand-drawing diagram element is a shape element of the diagram, the icon depicting the shape.

### Claim 20 (depends on 16)

A non-transitory computer readable medium as claimed in claim 16, wherein the first type element is text and depiction of the at least one recognized form of the hand-drawing diagram element is in typeset ink.
