# US 10,769,425 B2 · Method and system for extracting information from an image of a filled form document

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,425 B2
- Document ID: US10769425B2
- Status: Granted
- Kind: B2
- Application number: 16/101,763
- Filed: 2018-08-13
- Published: 2020-02-13
- Granted: 2020-09-08
- Earliest claimed: 2018-08-13
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Antonio Foncubierta Rodriguez; Maria Gabrani; Guillaume Jaume
- Web page: https://patentgrep.com/patent/US10769425B2

## CPC

- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/10: Character recognition
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of determining a hierarchy of a blank template using an image of the blank template and using the determined hierarchy for providing labels and field values of text lines of a filled form document.

## Claims

### Claim 1

A method of extracting information from an image of a filled form document, the form document comprising labeled fields comprising values and arranged in sections of the document, the method comprising:
providing an image of a blank template of the form document;
extracting first text lines from the image of the blank template using a text line recognizer of an optical character recognition (OCR) system;
performing using the OCR system an optical character recognition of the extracted first text lines resulting in first machine-encoded text lines;
merging the first machine encoded text lines into candidate sections based on the location of the first text lines in the blank template;
evaluating for each candidate section a set of predefined features indicative of the text lines and their location;
combining the evaluated features of each candidate section to generate an identifier of the candidate section, resulting in multiple identifiers;
determining sets of candidate sections that each share a respective identifier of the multiple identifiers;
selecting a set of candidate sections of the sets of sections that fulfills a predefined selection criterion based on the number of sections and number of respective fields;
determining as the hierarchy of the blank template the selected set of candidate sections;
extracting second text lines from the image of the filled form document using the text line recognizer;
performing using the OCR system an optical character recognition of the extracted second text lines resulting in second machine-encoded text lines;
merging the second machine encoded text lines into text regions based on the location of the second text lines;
for each second text line of the filled form document:
identifying the section of the determined hierarchy that matches the second text line based on the region of the second text line;
determining the first text line of the section that corresponds to the second text line by comparing the second text line with the first text lines, and determining the label of the field of the second text line and the position of the label within the filled form using the determined first text line;
using the determined labels for identifying and extracting field values of the second text lines;
comparing each of the field values with the determined labels for assigning the field value to a respective label of the determined labels.

### Claim 2 (depends on 1)

The method of claim 1, wherein comparing the first and second text lines is performed using a similarity metric indicative of the similarity between words of the first and second text lines.

### Claim 3 (depends on 2)

The method of claim 2, the similarity metric comprises at least one of Levenshtein distance, word accuracy (WA), combined word accuracy (CWA) and field detection accuracy (FDA).

### Claim 4 (depends on 1)

The method of claim 1, wherein the selection criterion is the following:
min
⁢
h
⁢

S

1

S

⁢
∑
i
=
0

S

-
1
⁢

Q
i

,
where S is the number of candidate sections shared by a given identifier and Q is the number of fields in a given candidate section.

### Claim 5 (depends on 1)

The method of claim 1, wherein the selection criterion is the following:
min
⁢
h
⁢

S

median
⁢
{

Q
}

,
where S is the number of candidate sections shared by a given identifier and median{Q} is the median number of fields for a given hierarchy.

### Claim 6 (depends on 1)

The method of claim 1, wherein the combining the evaluated features into an identifier h is performed as follows h=Σn=0Mfn·2n where fn represents the nth feature of M features.

### Claim 7 (depends on 1)

The method of claim 1, wherein the set of predefined features comprises at least one of: font size, font type, position of a character, background color, indication of a character being an uppercase or lowercase character.

### Claim 8 (depends on 1)

The method of claim 1, wherein the comparing of the field value with a label of the determined labels comprises computing a probability that the field value and the label corresponds to each other using visual features, wherein the visual features comprise L1 distance between centroids of bounding boxes of the compared label and field value, L1 distance between top-left corners of bounding boxes of the compared label and field value, L1 distance between closest borders of bounding boxes of the compared label and field value.

### Claim 9 (depends on 1)

The method of claim 1, wherein the merging of the machine encoded text lines is performed using a vertical merging technique.

### Claim 10 (depends on 1)

The method of claim 1, wherein the identifier of the text region comprises a hash value.

### Claim 11 (depends on 1)

The method of claim 1, further comprising providing the hierarchy of the filled form document as an xml file comprising relative positions of the text lines and sections.

### Claim 12 (depends on 1)

The method of claim 1, wherein the field value comprises a value that is input of the field and/or information indicative of the field.

### Claim 13 (depends on 1)

A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to implement steps of the method according to claim 1.

### Claim 14

A computer system for extracting information from an image of a filled form document, the form document comprising labeled fields comprising values and arranged in sections of the document, the computer system comprising an image of a blank template of the form document, the computer system being configured for:
extracting first text lines from the image of the blank template using a text line recognizer of an optical character recognition (OCR) system;
performing using the OCR system an optical character recognition of the extracted first text lines resulting in first machine-encoded text lines;
merging the first machine encoded text lines into candidate sections based on the location of the first text lines in the blank template;
evaluating for each candidate section a set of predefined features indicative of the text lines and their location;
combining the evaluated features of each candidate section to generate an identifier of the candidate section, resulting in multiple identifiers;
determining sets of candidate sections that each share a respective identifier of the multiple identifiers;
selecting a set of candidate sections of the sets of sections that fulfills a predefined selection criterion based on the number of sections and number of respective fields;
determining as the hierarchy of the blank template the selected set of candidate sections;
extracting second text lines from the image of the filled form document using the text line recognizer;
performing using the OCR system an optical character recognition of the extracted second text lines resulting in second machine-encoded text lines;
merging the second machine encoded text lines into text regions based on the location of the second text lines;
for each second text line of the filled form document:
identifying the section of the determined hierarchy that matches the second text line based on the region of the second text line;
determining the first text line of the section that corresponds to the second text line by comparing the second text line with the first text lines, and determining the label of the field of the second text line and the position of the label within the filled form using the determined first text line;
using the determined labels for identifying and extracting field values of the second text lines;
comparing each of the field values with the determined labels for assigning the field value to a respective label of the determined labels.