# US 10,482,170 B2 · User interface for contextual document recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,170 B2
- Document ID: US10482170B2
- Status: Granted
- Kind: B2
- Application number: 15/785,655
- Filed: 2017-10-17
- Published: 2019-04-18
- Granted: 2019-11-19
- Earliest claimed: 2017-10-17
- Assignees: HRB Innovations, Inc.
- Applicants: HRB Innovations, Inc.
- Inventors: Eric R. Roebuck; Mary Clark; Erik V. Lustig; David Jason Gregory Butler
- Web page: https://patentgrep.com/patent/US10482170B2

## CPC

- G06F 40/174: Form filling; Merging
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/10: Character recognition
- G06V 30/153: {using recognition of characters or words}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Media and method for presenting a user interface for verifying automatically recognized information from forms. Embodiments of the invention may import an image of a form and break the form down into a series of bounding boxes, each including the value of a form item and a description and/or label for the form item. When making use of a particular value from the form image, embodiments of the invention may present the bounding box containing that value to the user, together with enough context for the user to verify that the correct item has been identified and that the automatically recognized value is correct.

## Claims

### Claim 1

A method of presenting a user interface for verifying automatically recognized information, comprising the steps of:
receiving a form image of a form, the form including plurality of values and a corresponding plurality of descriptions for the plurality of values;
identifying a plurality of regions of the form image corresponding to the plurality of values,
wherein each region of the form image contains an image of a value and a description corresponding to the region;
performing automatic recognition of each region to obtain automatically recognized values and automatically recognized descriptions of each region;
determining if the automatically recognized values are correct or incorrect based on a comparison of the automatically recognized values for each region with the form image after automatic recognition;
determining a needed value based on the comparison of the form image and the automatically recognized regions;
identifying a region of the automatically recognized regions corresponding to the needed value,
wherein the identified region includes an automatically recognized value and an automatically recognized description of the identified region;
presenting a user interface to the user comprising:
the plurality of regions of the form image;
the identified region of the form image corresponding to the needed value,
wherein the identified region is automatically highlighted prior to input from the user;
the automatically recognized value for the identified region;
the automatically recognized description for the identified region;
a first control allowing the user to accept the automatically recognized value,
wherein a plurality of regions of the form image that do not correspond to the need value are not presented via the user interface.

### Claim 2 (depends on 1)

The method of claim 1, wherein the identified region is identified based on an automatically identified label for the region.

### Claim 3 (depends on 2)

The method of claim 2, further comprising the step of presenting, in the user interface, the automatically identified label for the region.

### Claim 4 (depends on 1)

The method of claim 1, further comprising the steps of presenting, in the user interface:
a second control allowing the user to reject the automatically recognized value; and
a third control allowing the user to provide a corrected value for the identified region.

### Claim 5 (depends on 1)

The method of claim 1, further comprising the steps of presenting, in the user interface:
a second control allowing the user to reject the identified region; and
a third control allowing the user to provide a corrected region of the form image corresponding to the needed value.

### Claim 6 (depends on 5)

The method of claim 5, further comprising the steps of presenting, in the user interface:
an automatically recognized value for the corrected region; and
a fourth control allowing the user to accept the automatically recognized value for the corrected region as correct.

### Claim 7 (depends on 1)

The method of claim 1, wherein the identified region is presented as an emphasized region in the form image.

### Claim 8

One or more non-transitory computer-readable media storing computer-executable instructions for a method of presenting a user interface for verifying automatically recognized information that, when executed:
receive a form image of a form, the form including plurality of values and a corresponding plurality of labels for the plurality of values;
identifying a plurality of regions of the form image corresponding to the plurality of values,
wherein each region of the form image contains an image of a value and a label corresponding to the region;
for each of the plurality of regions, perform automatic recognition of the value and the label for the region to obtain an automatically recognized value for the region and an automatically recognized label for the region;
determining if the automatically recognized value is correct or incorrect based on a comparison of the automatically recognized value for the region with the form image after automatic recognition;
determining a needed value based on the comparison;
identify a region of the plurality of regions of the form image corresponding to the needed value, and
presenting a user interface to the user comprising:
the plurality of regions of the form image;
the identified region of the form image corresponding to the needed value,
wherein the identified region is automatically magnified prior to input by the user;
an automatically recognized value for the identified region;
an automatically recognized label for the identified region;
a first control allowing the user to accept the automatically recognized value and an automatically recognized label.

### Claim 9 (depends on 8)

The media of claim 8, wherein the user interface further comprises:
a second control allowing the user to reject the automatically recognized value; and
a third control allowing the user to provide a corrected value for the identified region.

### Claim 10 (depends on 8)

The media of claim 8, wherein the user interface further comprises:
a second control allowing the user to reject the identified region; and
a third control allowing the user to provide a corrected region of the form image corresponding to the needed value.

### Claim 11 (depends on 10)

The media of claim 10, wherein the computer-executable instructions, when executed, further:
perform automatic recognition of a value for the corrected region; and
present, in the user interface:
the automatically recognized value for the corrected region; and
a fourth control allowing the user to accept the automatically recognized value for the corrected region as correct.

### Claim 12 (depends on 8)

The media of claim 8, wherein the step of performing automatic recognition of the value for the region and the label for the region comprises using optical character recognition to determine at least one of the value for the region and the label for the region.

### Claim 13 (depends on 8)

The media of claim 8, wherein the step of performing automatic recognition of the value for the region and the label for the region further comprises preforming automatic recognition of a description for the value.

### Claim 14 (depends on 8)

The media of claim 8, wherein the identified region of the form corresponding to the needed region is presented with a surrounding portion of the form image.

### Claim 15

A system comprising:
a user interface comprising:
an identified region of a form image corresponding to a needed value;
an automatically recognized value for the identified region;
a first control allowing a user to accept the automatically recognized value,
wherein a plurality of regions of the form image that do not correspond to a needed value are not presented via the user interface; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed perform a method of presenting a user interface for verifying automatically recognized information, the method comprising the steps of:
identifying the plurality of regions of the form image of a form, wherein the form includes a plurality of values, and wherein each region of the plurality of regions corresponds to a value of the plurality of values;
performing automatic recognition of the plurality of regions of the form image to obtain automatically recognized values;
determining if the automatically recognized values are correct or incorrect based on a comparison of the automatically recognized values for each region with the form image after automatic recognition;
determining the needed value based on the comparison of the form image and the automatically recognized plurality of regions;
identifying a region of the plurality of regions of the form image corresponding to the needed value.

### Claim 16 (depends on 15)

The system of claim 15, wherein the method further comprises the steps of presenting, in the user interface:
a second control allowing the user to reject the automatically recognized value; and
a third control allowing the user to provide a corrected value for the identified region.

### Claim 17 (depends on 15)

The system of claim 15, further comprising the steps of presenting in the user interface:
a second control allowing the user to reject the identified region; and
a third control allowing the user to provide a corrected region of the form image corresponding to the needed value.

### Claim 18 (depends on 17)

The system of claim 17, wherein the method further comprises the steps of presenting, in the user interface:
an automatically recognized value for the corrected region; and
a fourth control allowing the user to accept the automatically recognized value for the corrected region as correct.

### Claim 19 (depends on 15)

The system of claim 15, wherein the method further comprises the step of presenting, in the user interface, an automatically recognized label for the automatically identified value.

### Claim 20 (depends on 15)

The system of claim 15, wherein the method further comprises the step of presenting, in the user interface, an automatically recognized description for the automatically recognized value.