# US 2021/0279458 A1 · Interactive editing of electronic text

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279458 A1
- Document ID: US20210279458A1
- Status: Published application
- Kind: A1
- Application number: 17/327,034
- Filed: 2021-05-21
- Published: 2021-09-09
- Earliest claimed: 2019-09-16
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Maria Cirimele; Thomas William Buckley; Robert Ky Mickle; Tayeb Al Karim
- Web page: https://patentgrep.com/patent/US20210279458A1

## CPC

- G06V 30/32: Digital ink
- G06F 3/03545: {Pens or stylus}
- G06F 40/171: by use of digital ink
- G06V 30/153: {using recognition of characters or words}
- G06V 30/347: {Sampling; Contour coding; Stroke extraction}
- G06V 30/36: {Matching; Classification}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A new segment of electronic handwriting is provided to a handwriting recognition module to obtain a plurality of textual interpretations of the new segment. The textual interpretations obtained from the handwriting recognition module are scored based on how each respective electronic handwriting representation would change a display of existing electronic content when the respective electronic handwriting representation is displayed substantially at the user designated position within or adjacent to the existing electronic content. Based on the scoring, an electronic handwriting representation corresponding to a respective textual interpretation of the plurality of textual interpretations is selected, and the existing electronic content is modified to include the selected electronic handwriting representation located substantially at the user designated position.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method comprising:
determining, by a computing system and based on a segment of electronic handwriting, a plurality of candidate content segments;
generating, by the computing system and based on the plurality of candidate content segments, a respective electronic handwriting representation for each candidate content segment from the plurality of candidate content segments;
determining, by the computing system, a respective score for each of the plurality of candidate content segments based on a change in a display of existing electronic content if the respective electronic handwriting representation is displayed in place of the segment of the electronic handwriting;
selecting, by the computing system and based on the respective scores, one of the plurality of candidate content segments as a selected electronic handwriting representation; and
outputting, for display on a display device, the selected electronic handwriting representation.

### Claim 22 (depends on 21)

The method of claim 21, wherein the electronic handwriting representations includes one or more computer-replicated glyphs substantially in a handwriting style of the existing electronic content, the method further comprising:
dynamically reflowing at least a portion of the existing electronic content to accommodate the selected electronic handwriting representation.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
determining, by the computing system, that a score associated with the selected electronic handwriting representation does not satisfy a predetermined confidence level; and
outputting, for display on the display device, a visual indication that the selected electronic handwriting representation does not satisfy the predetermined confidence level.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
before outputting the selected electronic handwriting representation, outputting, for display on the display device, a visual indication of a computer designated position within or adjacent to the existing electronic content corresponding to an insertion point of the selected electronic handwriting representation;
receiving, by the computing system, a user input indicating user acceptance or rejection of the selected electronic handwriting representation; and
responsive to receiving an indication of user rejection of the selected electronic handwriting representation, selecting, by the computing system, a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation.

### Claim 25 (depends on 21)

The method of claim 21, further comprising:
maintaining, in a memory of the computing system, a character-based textual representation of the existing electronic content displayed on the display device; and
storing, in the memory of the computing system, an updated character-based textual representation that includes a textual interpretation of the selected electronic handwriting representation.

### Claim 26 (depends on 25)

The method of claim 25, further comprising:
outputting, for display on the display device, the updated character-based textual representation, the updated character-based textual representation including a plurality of words, each word from the plurality of words being positioned adjacent to or directly underneath a corresponding word segment of the existing electronic content.

### Claim 27 (depends on 21)

The method of claim 21, further comprising:
receiving, by the computing system, an indication that the selected electronic handwriting representation was rejected by a user;
selecting, by the computing system, a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation; and
outputting, for display on the display device, updated electronic content that includes the updated selected electronic handwriting representation substantially at a user designated position.

### Claim 28 (depends on 27)

The method of claim 27, further comprising:
determining, by the computing system, a respective updated score for each of the plurality of candidate content segments based on user input associated with the indication that the selected electronic handwriting representation was rejected and based on how each respective electronic handwriting representation would change the display of the existing electronic content when the respective electronic handwriting representation is displayed substantially at the user designated position within or adjacent to the existing electronic content; and
selecting, by the computing system and based on the respective updated scores, a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation.

### Claim 29 (depends on 21)

The method of claim 21, wherein the segment of electronic handwriting modifies an existing segment of electronic handwriting of the existing electronic content, and wherein the existing segment of electronic handwriting is provided with the segment to a handwriting recognition module to obtain the plurality of candidate content segments.

### Claim 30 (depends on 21)

The method of claim 21, further comprising:
generating a tree of content actions for the segment of electronic handwriting, each lower node that branches from a higher level node in the tree being representative of one of a plurality of content actions that can be performed with respect to the existing electronic content to identify a portion of the existing electronic content for association with at least a portion of the segment of electronic handwriting, the identified portion of the existing electronic content being delineated according to a number of nodes along a given branch, determining, automatically and without user intervention, candidate portions of the existing electronic content to provide to a handwriting recognition module with the segment of electronic handwriting based on automatically traversing the tree along respective paths from a root node of the tree to respective leaf nodes of the tree and identifying the portions of the existing electronic content associated with the content actions represented by nodes along the path; and
providing the segment of electronic handwriting with each determined candidate portion of the existing electronic content to the handwriting recognition module to obtain the plurality of textual interpretations of the segment of electronic handwriting.

### Claim 31

A system comprising:
one or more processors;
a touch-sensitive display device; and
a memory storage device storing instructions that, when executed by the one or more processors, cause the one or more processors to:
determine, based on a segment of electronic handwriting, a plurality of candidate content segments;
generate, based on the plurality of candidate content segments, a respective electronic handwriting representation for each candidate content segment from the plurality of candidate content segments;
determine a respective score for each of the plurality of candidate content segments based on a change in a display of existing electronic content if the respective electronic handwriting representation is displayed in place of the segment of the electronic handwriting;
select, based on the respective scores, one of the candidate content segments as a selected electronic handwriting representation; and
output, for display by the touch-sensitive display device, the selected electronic handwriting representation.

### Claim 32 (depends on 31)

The system of claim 31, wherein the instructions further cause the one or more processors to:
determine that a score associated with the selected electronic handwriting representation does not satisfy a predetermined confidence level; and
output, for display by the touch-sensitive display device, a visual indication that the selected electronic handwriting representation does not satisfy the predetermined confidence level.

### Claim 33 (depends on 32)

The system of claim 32, wherein the instructions further cause the one or more processors to:
before outputting the selected electronic handwriting representation, output, for display by the touch-sensitive display device, a visual indication of a computer designated position within or adjacent to the existing electronic content corresponding to an insertion point of the selected electronic handwriting representation; and
responsive to receiving an indication of a user input rejecting the selected electronic handwriting representation, select a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation.

### Claim 34 (depends on 31)

The system of claim 31, wherein the instructions further cause the one or more processors to:
maintain, in the memory storage device, a character-based textual representation of the existing electronic content displayed on the display device;
store, in the memory storage device, an updated character-based textual representation that includes a textual interpretation of the selected electronic handwriting representation; and
output, for display by the touch-sensitive display device, the updated character-based textual representation, the updated character-based textual representation including a plurality of words, each word from the plurality of words being positioned adjacent to or directly underneath a corresponding word segment of the existing electronic content.

### Claim 35 (depends on 31)

The system of claim 31, wherein the instructions further cause the one or more processors to:
receive an indication that the selected electronic handwriting representation was rejected by a user;
determine a respective updated score for each of the plurality of candidate content segments based on user input associated with the indication that the selected electronic handwriting representation was rejected and based on how each respective electronic handwriting representation would change the display of the existing electronic content when the respective electronic handwriting representation is displayed substantially at a user designated position within or adjacent to the existing electronic content;
select, based on the respective updated scores, a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation; and
output, for display by the touch-sensitive display device, updated electronic content that includes the updated selected electronic handwriting representation substantially at a user designated position.

### Claim 36

A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a computing device, cause the one or more processors to:
determine, based on a segment of electronic handwriting, a plurality of candidate content segments;
generate, based on the plurality of candidate content segments, a respective electronic handwriting representation for each candidate content segment from the plurality of candidate content segments;
determine a respective score for each of the plurality of candidate content segments based on a change in a display of existing electronic content if the respective electronic handwriting representation is displayed in place of the segment of the electronic handwriting;
select, based on the respective scores, one of the candidate content segments as a selected electronic handwriting representation; and
output, for display by a display device, the selected electronic handwriting representation.

### Claim 37 (depends on 36)

The non-transitory computer-readable medium of claim 36, wherein the instructions further cause the one or more processors to:
determine that a score associated with the selected electronic handwriting representation does not satisfy a predetermined confidence level; and
output, for display by the display device, a visual indication that the selected electronic handwriting representation does not satisfy the predetermined confidence level.

### Claim 38 (depends on 37)

The non-transitory computer-readable medium of claim 37, wherein the instructions further cause the one or more processors to:
before outputting the selected electronic handwriting representation, output, for display by the display device, a visual indication of a computer designated position within or adjacent to the existing electronic content corresponding to an insertion point of the selected electronic handwriting representation; and
responsive to receiving an indication of a user input rejecting the selected electronic handwriting representation, select a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation.

### Claim 39 (depends on 36)

The non-transitory computer-readable medium of claim 36, wherein the instructions further cause the one or more processors to:
maintain, in a memory, a character-based textual representation of the existing electronic content displayed on the display device;
store, in the memory, an updated character-based textual representation that includes a textual interpretation of the selected electronic handwriting representation; and
output, for display by the display device, the updated character-based textual representation, the updated character-based textual representation including a plurality of words, each word from the plurality of words being positioned adjacent to or directly underneath a corresponding word segment of the existing electronic content.

### Claim 40 (depends on 36)

The non-transitory computer-readable medium of claim 36, wherein the instructions further cause the one or more processors to:
receive an indication that the selected electronic handwriting representation was rejected by a user;
determine a respective updated score for each of the plurality of candidate content segments based on user input associated with the indication that the selected electronic handwriting representation was rejected and based on how each respective electronic handwriting representation would change the display of the existing electronic content when the respective electronic handwriting representation is displayed substantially at a user designated position within or adjacent to the existing electronic content;
select, based on the respective updated scores, a different one of the plurality of candidate content segments as an updated selected electronic handwriting representation; and
output, for display by the display device, updated electronic content that includes the updated selected electronic handwriting representation substantially at a user designated position.
