# US 2022/0044058 A1 · Template-Based Key-Value Extraction for Inferring OCR Key Values Within Form Images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044058 A1
- Document ID: US20220044058A1
- Status: Published application
- Kind: A1
- Application number: 16/988,536
- Filed: 2020-08-07
- Published: 2022-02-10
- Earliest claimed: 2020-08-07
- Assignees: salesforce.com, inc.
- Applicants: salesforce.com, inc.
- Inventors: Shu Zhang; Chetan Ramaiah; Ran Xu; Caiming Xiong
- Web page: https://patentgrep.com/patent/US20220044058A1

## CPC

- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/10: Character recognition
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The system has a form analysis module that receives an image of a form into which values have been filled for the possible fields of information on the form, such as first name, address, age, and the like. By using a library of form templates, a form analysis module allows both flexibility of form processing and simplicity for the user. That is, the techniques used by the form analysis module allow the processing of any form image for which the library has a form template example. The form image need not precisely match any form template, but rather may be scaled or shifted relative to a corresponding template. Additionally, the user need only provide the form image itself, without providing any additional exemplars, metadata for training, or the like.

## Claims

### Claim 1

A computer-implemented method for location of text of key-value pairs on a form image, the computer-implemented method comprising:
obtaining, at one or more processors, a form image originating from a client device;
identifying, by the one or more processors using a neural network, regions of text within the form image;
selecting, by the one or more processors, a form template from a database storing a plurality of form templates according to the identified regions of text;
computing, by the one or more processors, a spatial mapping between the selected form template and the obtained form image;
computing, by the one or more processors, spatial relationships between key regions and value regions of the selected form template;
determining, by the one or more processors, for each key of a plurality of keys of the selected form template, using the computed spatial mappings and spatial relationships, an estimated region on the form image for the value corresponding to the key; and
extracting, by the one or more processors, from the estimated regions on the form image based on overlap of the identified text regions and the estimated regions, text corresponding to values of the plurality of keys of the selected form template.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising determining a template type for the obtained template by using a model trained by a machine learning algorithm.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising obtaining, from the client device, an indicator of a template type for the obtained template.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, the form template being selected in part using the obtained template type.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, the selecting the form template comprising matching locations of the identified regions of text with locations in which the form templates indicate that text is to be found.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, the selecting the form template comprising matching key labels specified by the form templates with text form the identified regions of text.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the spatial mapping is a linear mapping specifying a scaling component and a shift component.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the spatial relationships comprise at least one of a horizontal gap or a vertical gap between a key label region and a corresponding value region.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein determining an estimated region on the form image for the value corresponding to a key comprises:
responsive to determining that the key has label text on the form image, determining the estimated region using the determined spatial relationship for the key.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein determining the estimated regions on the form image for the value corresponding to the key comprises:
responsive to determining that a key lacks label text on the form image, determining the estimated region using the determined spatial mapping.

### Claim 11

A non-transitory computer-readable storage medium containing instructions that when executed by a computer processor perform actions comprising:
obtaining, at one or more processors, a form image originating from a client device;
identifying, by the one or more processors using a neural network, regions of text within the form image;
selecting, by the one or more processors, a form template from a database storing a plurality of form templates according to the identified regions of text;
computing, by the one or more processors, a spatial mapping between the selected form template and the obtained form image;
computing, by the one or more processors, spatial relationships between key regions and value regions of the selected form template;
determining, by the one or more processors, for each key of a plurality of keys of the selected form template, using the computed spatial mappings and spatial relationships, an estimated region on the form image for the value corresponding to the key; and
extracting, by the one or more processors, from the estimated regions on the form image based on overlap of the identified text regions and the estimated regions, text corresponding to values of the plurality of keys of the selected form template.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, the actions further comprising determining a template type for the obtained template by using a model trained by a machine learning algorithm.

### Claim 13 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, the actions further comprising obtaining, from the client device, an indicator of a template type for the obtained template.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, the form template being selected in part using the obtained template type.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, the selecting the form template comprising matching locations of the identified regions of text with locations in which the form templates indicate that text is to be found.

### Claim 16 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, the selecting the form template comprising matching key labels specified by the form templates with text form the identified regions of text.

### Claim 17 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the spatial mapping is a linear mapping specifying a scaling component and a shift component.

### Claim 18 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the spatial relationships comprise at least one of a horizontal gap or a vertical gap between a key label region and a corresponding value region.

### Claim 19 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining an estimated region on the form image for the value corresponding to a key comprises:
responsive to determining that the key has label text on the form image, determining the estimated region using the determined spatial relationship for the key.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein determining the estimated regions on the form image for the value corresponding to the key comprises:
responsive to determining that a key lacks label text on the form image, determining the estimated region using the determined spatial mapping.
