# US 2019/0171873 A1 · Cognitive document image digitization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171873 A1
- Document ID: US20190171873A1
- Status: Published application
- Kind: A1
- Application number: 15/829,304
- Filed: 2017-12-01
- Published: 2019-06-06
- Earliest claimed: 2017-12-01
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Kevin NORTHRUP; Craig M. TRIM; Badr KHAMIS; Karan SEHGAL; Abisola ADENIRAN; Chandrashekhar PADOLE
- Web page: https://patentgrep.com/patent/US20190171873A1

## CPC

- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 30/10: Character recognition
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/43: {Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, computer program products, and systems are presented. The methods include, for instance: obtaining a document image with objects and identifying microblocks corresponding to each object. Analyzing a position of a microblock for collinearity with another microblock based on respective positional characteristics and adjustable collinearity parameters. Collinear microblocks are identified into a macroblock and computational data of a key-value pair is created from the macroblock. A heuristic confidence level is associated with the key-value pair. Also based on data cluster formation, a table may be classified and data extracted.

## Claims

### Claim 1

A computer implemented method for extracting data from a document image comprising:
obtaining, by one or more processor of a computer, the document image, wherein the document image includes a plurality of objects;
identifying a plurality of microblocks respectively corresponding to the plurality of objects, wherein each of the microblocks is associated with a content, a position, and a style;
discovering that a first microblock is collinear with a second microblock based on respective positional characteristics of the first microblock and the second microblock and adjustable collinearity parameters;
identifying a macroblock with the first microblock and the second microblock, wherein the macroblock constitutes a key-value pair that is computational by computer programs;
associating a confidence level with the key-value pair based on heuristics; and
communicating the key-value pair and the associated confidence level to a user.

### Claim 2 (depends on 1)

The computer implemented method of claim 1, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
ascertaining that the first microblock and the second microblock are either horizontally or vertically aligned; and
determining that the first microblock and the second microblock are collinear.

### Claim 3 (depends on 1)

The computer implemented method of claim 1, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
ascertaining that both the first microblock and the second microblock are enclosed in a certain geometric area that does not enclose another microblock; and
determining that the first microblock and the second microblock are collinear.

### Claim 4 (depends on 1)

The computer implemented method of claim 1, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
analyzing respective semantic contents of both the first microblock and the second microblock, wherein the semantic contents is a member of the adjustable collinearity parameters;
ascertaining that a first semantic content of the first microblock is associated with a key name;
discovering, from key ontology data corresponding to the key name, that a second semantic content of the second microblock is of a data type corresponding to the key name; and
determining that the first microblock and the second microblock are collinear.

### Claim 5 (depends on 1)

The computer implemented method of claim 1, wherein the style of each microblock includes a font type, a font size, a paragraph alignment, and zero or more punctuation delimiter of said each microblock, and wherein the adjustable collinearity parameters include the style of said each microblock.

### Claim 6 (depends on 1)

The computer implemented method of claim 1, further comprising:
ascertaining a document class for the document image;
determining that a class key specified in the document class is missing from a set of macroblocks for the document image;
searching respective contents of the microblocks from the identified macroblocks for an alias corresponding to the missing class key;
selecting another macroblock associated one of the microblocks having the alias as content; and
identifying another key-value pair with the selected another macroblock in place of a key-value pair for the missing class key.

### Claim 7

A computer implemented method for extracting data from a document image comprising:
obtaining, by one or more processor of a computer, the document image, wherein the document image includes a tabular object cluster;
identifying a plurality of macroblocks in the tabular object cluster, wherein respective macroblocks include collinear objects;
locating, within each macroblock, one or more candidate cells from respective microblocks;
classifying the tabular object cluster, based on the located candidate cells, into a table class from a predefined set of table classes;
extracting, according to the table class, data from the one or more candidate cells for each macroblock, wherein the extracted data is computational by computer programs; and
creating one or more two-dimensional (2D) array from the extracted data, wherein the one or more 2D array has a number of macroblocks in the tabular object cluster for a first dimension.

### Claim 8 (depends on 7)

The computer implemented method of claim 7, wherein a macroblock of the tabular object cluster corresponds to a column of a table, and wherein columns may have respective headings in a same row of each column.

### Claim 9 (depends on 7)

The computer implemented method of claim 7, the identifying comprising:
determining each macroblock from the plurality of macroblocks by analyzing column definition attributes of two or more groups of vertically collinear objects from the tabular object cluster, wherein the column definition attributes comprises absolute distances between any two adjacent macroblocks, relative distances between the any two adjacent macroblocks, semantic interpolation, and relative styles respective to each macroblock.

### Claim 10 (depends on 7)

The computer implemented method of claim 7, the locating comprising:
sequencing respective microblocks in each macroblock of the tabular object cluster by assigning respective sequence number to the respective microblocks within each macroblock, wherein each microblock corresponds to a block of one or more word represented in characters and/or symbols; and
analyzing positional relationships amongst microblocks having the same sequence number across all macroblocks.

### Claim 11 (depends on 7)

The computer implemented method of claim 7, the classifying comprising:
assigning the table class to the tabular object cluster based on positional relationships amongst the identified macroblocks and the located candidate cells, wherein the predefined set of table classes includes a first table class of tables with merged cells, a second table class of tables with indefinite cell borders, and a third table class of tables with nested cells.

### Claim 12 (depends on 7)

The computer implemented method of claim 7, the extracting comprising:
creating a key-value pair corresponding to a candidate cell, wherein a key of the key-value pair a heading of a macroblock from which the candidate cell locates, wherein the heading of the macroblock is associated with a confidence level indicating a likelihood of the heading being a correct key for the candidate cell.

### Claim 13

A computer program product comprising:
a computer readable storage medium readable by one or more processor and storing instructions for execution by the one or more processor for performing a method for extracting data from a document image comprising:
obtaining the document image, wherein the document image includes a plurality of objects;
identifying a plurality of microblocks respectively corresponding to the plurality of objects, wherein each of the microblocks is associated with a content, a position, and a style;
discovering that a first microblock is collinear with a second microblock based on respective positional characteristics of the first microblock and the second microblock and adjustable collinearity parameters;
identifying a macroblock with the first microblock and the second microblock, wherein the macroblock constitutes a key-value pair that is computational by computer programs;
associating a confidence level with the key-value pair based on heuristics; and
communicating the key-value pair and the associated confidence level to a user.

### Claim 14 (depends on 13)

The computer program product of claim 13, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
ascertaining that the first microblock and the second microblock are either horizontally or vertically aligned; and
determining that the first microblock and the second microblock are collinear.

### Claim 15 (depends on 13)

The computer program product of claim 13, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
ascertaining that both the first microblock and the second microblock are enclosed in a certain geometric area that does not enclose another microblock; and
determining that the first microblock and the second microblock are collinear.

### Claim 16 (depends on 13)

The computer program product of claim 13, the discovering comprising:
ascertaining that the first microblock and the second microblock are immediately adjacent with each other based on analyzing the respective positional characteristics of the first microblock and the second microblock;
analyzing respective semantic contents of both the first microblock and the second microblock, wherein the semantic contents is a member of the adjustable collinearity parameters;
ascertaining that a first semantic content of the first microblock is associated with a key name;
discovering, from key ontology data corresponding to the key name, that a second semantic content of the second microblock is of a data type corresponding to the key name; and
determining that the first microblock and the second microblock are collinear.

### Claim 17 (depends on 13)

The computer program product of claim 13, wherein the style of each microblock includes a font type, a font size, a paragraph alignment, and zero or more punctuation delimiter of said each microblock, and wherein the adjustable collinearity parameters include the style of said each microblock.

### Claim 18 (depends on 13)

The computer program product of claim 13, further comprising:
ascertaining a document class for the document image;
determining that a class key specified in the document class is missing from a set of macroblocks for the document image;
searching respective contents of the microblocks from the identified macroblocks for an alias corresponding to the missing class key;
selecting another macroblock associated one of the microblocks having the alias as content; and
identifying another key-value pair with the selected another macroblock in place of a key-value pair for the missing class key.

### Claim 19

A computer program product comprising:
a computer readable storage medium readable by one or more processor and storing instructions for execution by the one or more processor for performing a method for extracting data from a document image comprising:
obtaining the document image, wherein the document image includes a tabular object cluster;
identifying a plurality of macroblocks in the tabular object cluster, wherein respective macroblocks include collinear objects;
locating, within each macroblock, one or more candidate cells from respective microblocks;
classifying the tabular object cluster, based on the located candidate cells, into a table class from a predefined set of table classes;
extracting, according to the table class, data from the one or more candidate cells for each macroblock, wherein the extracted data is computational by computer programs; and
creating one or more two-dimensional (2D) array from the extracted data, wherein the one or more 2D array has a number of macroblocks in the tabular object cluster for a first dimension.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein a macroblock of the tabular object cluster corresponds to a column of a table, and wherein columns may have respective headings in a same row of each column.

### Claim 21 (depends on 19)

The computer program product of claim 19, the identifying comprising:
determining each macroblock from the plurality of macroblocks by analyzing column definition attributes of two or more groups of vertically collinear objects from the tabular object cluster, wherein the column definition attributes comprises absolute distances between any two adjacent macroblocks, relative distances between the any two adjacent macroblocks, semantic interpolation, and relative styles respective to each macroblock.

### Claim 22 (depends on 19)

The computer program product of claim 19, the locating comprising:
sequencing respective microblocks in each macroblock of the tabular object cluster by assigning respective sequence number to the respective microblocks within each macroblock, wherein each microblock corresponds to a block of one or more word represented in characters and/or symbols; and
analyzing positional relationships amongst microblocks having the same sequence number across all macroblocks.

### Claim 23 (depends on 19)

The computer program product of claim 19, the classifying comprising:
assigning the table class to the tabular object cluster based on positional relationships amongst the identified macroblocks and the located candidate cells, wherein the predefined set of table classes includes a first table class of tables with merged cells, a second table class of tables with indefinite cell borders, and a third table class of tables with nested cells.

### Claim 24 (depends on 19)

The computer program product of claim 19, the extracting comprising:
creating a key-value pair corresponding to a candidate cell, wherein a key of the key-value pair a heading of a macroblock from which the candidate cell locates, wherein the heading of the macroblock is associated with a confidence level indicating a likelihood of the heading being a correct key for the candidate cell.

### Claim 25

A system comprising:
a memory;
one or more processor in communication with memory; and
program instructions executable by the one or more processor via the memory to perform a method for extracting data from a document image comprising:
obtaining the document image, wherein the document image includes a plurality of objects;
identifying a plurality of microblocks respectively corresponding to the plurality of objects, wherein each of the microblocks is associated with a content, a position, and a style;
discovering that a first microblock is collinear with a second microblock based on respective positional characteristics of the first microblock and the second microblock and adjustable collinearity parameters;
identifying a macroblock with the first microblock and the second microblock, wherein the macroblock constitutes a key-value pair that is computational by computer programs;
associating a confidence level with the key-value pair based on heuristics; and
communicating the key-value pair and the associated confidence level to a user.
