# US 12,217,523 B2 · List and tabular data extraction system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,523 B2
- Document ID: US12217523B2
- Status: Granted
- Kind: B2
- Application number: 17/898,193
- Filed: 2022-08-29
- Published: 2023-12-21
- Granted: 2025-02-04
- Earliest claimed: 2022-06-14
- Assignees: Accenture Global Solutions Limited
- Applicants: Accenture Global Solutions Limited
- Inventors: Andre Chatzistamatiou; Florin Cremenescu; Yizhen Dai; Ludo Gerardus Wilhelmus van Alst
- Web page: https://patentgrep.com/patent/US12217523B2

## CPC

- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 16/906: Clustering; Classification
- G06F 16/93: Document management systems
- G06F 18/20: Analysing
- G06F 18/24: Classification techniques
- G06F 40/106: Display of layout of documents; Previewing
- G06F 40/177: of tables; using ruled lines
- G06F 40/258: Heading extraction; Automatic titling; Numbering
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 30/10: Character recognition
- G06V 30/1463: {Orientation detection or correction, e.g. rotation of multiples of 90 degrees}
- G06V 30/1475: {Inclination or skew detection or correction of characters or of image to be recognised}
- G06V 30/153: {using recognition of characters or words}
- G06V 30/162: Quantising the image signal
- G06V 30/164: Noise filtering
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for automating and improving tabular and list-based data extraction from a variety of document types is disclosed. The system and method detect and sort which documents include tables or lists, and performs row and column segmentation. In addition, the system and method apply Conditional Random Fields models to localize each table and semantic data understanding to map and export the extracted data to the desired format and arrangement.

## Claims

### Claim 1

A computer-implemented method of tabular or list-based data extraction from document images, the method comprising:
receiving, at a server and from a first data source, a first document including a first page;
performing a column-wise pixel analysis of the first page, thereby determining that the first page includes a first table;
performing column segmentation based on signal analysis of column-wise mean pixel values of the first page, thereby identifying a set of columns;
performing row segmentation using optical character recognition (OCR)-generated bounding boxes, thereby identifying a set of rows;
selecting which rows of the set of rows belong to the first table using a first Conditional Random Fields (CRF) model, thereby localizing the first table on the first page;
selecting, for each column in the set of columns, a header name from a pre-defined set of header names, the selection being based on a classification performed by a second CRF model that evaluates at least the entire contents of that column;
mapping each item of data extracted from a cell in the first table to a field using semantic data understanding; and
generating a first digital table representing data extracted from the first table for presentation in a user interface.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
performing image binarization of the first page to produce a binarized page, where the binarized page is at a first orientation;
selecting a first range of rotation including a first number of angles extending in a sequence from a first angle to a second angle, the angles in the sequence being evenly spaced apart by a first increment;
incrementally rotating the binarized page to align with each angle in the first range of rotation such that the binarized page rotates step-wise from the first orientation until oriented at a second orientation associated with the second angle;
calculating, for each incremental rotation of the binarized page through the first range of rotation, a set of row-wise mean pixel values of the binarized page, thereby producing a plurality of row-wise mean pixel value sets, the plurality of row-wise mean pixel value sets including a first set of row-wise mean pixel values for the binarized page while at a third orientation associated with a third angle of the first range of rotation;
generating, for each orientation of the binarized page resulting from the incremental rotation through the first range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a first signal representing the first set of row-wise mean pixel values;
calculating, for each signal in the plurality of signals, a variance level of that signal;
determining the first signal has the highest variance level; and
selecting, for further processing, the binarized image in the third orientation based on the first signal having the highest variance level.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
selecting a second range of rotation including a second number of angles extending in a sequence from a fourth angle to a fifth angle, the angles in the sequence being evenly spaced apart by a second increment that is smaller than the first increment;
incrementally rotating the binarized page to align with each angle in the second range of rotation;
calculating, for each incremental rotation of the binarized page through the second range of rotation, a set of row-wise mean pixel values of the binarized page;
generating, for each orientation of the binarized page resulting from the incremental rotation through the second range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a second signal for the binarized page in a fourth orientation; and
determining the binarized image in the fourth orientation is less skewed than the binarized image in the third orientation if a variance level for the second signal is greater than the variance level for the first signal.

### Claim 4 (depends on 3)

The method of claim 3, wherein the second range of rotation includes the third angle.

### Claim 5 (depends on 2)

The method of claim 2, further comprising employing a minmax-concave total variation denoising technique to smooth the first signal.

### Claim 6 (depends on 1)

The method of claim 1, further comprising classifying each row into one of three classes comprising a header row, a table row, and an other row, and wherein a position of the first table on the first page is determined based on the classification of each row into one of the three classes.

### Claim 7 (depends on 1)

The method of claim 1, wherein the set of columns includes a first column and a directly adjacent second column, and the method further comprises:
determining, using semantic data understanding, that first data in the first column and second data in the second column have a higher likelihood of representing meaningful content when evaluated as a single data entry; and
merging the first column and the second column into a single, third column.

### Claim 8 (depends on 1)

The method of claim 1, wherein the set of columns includes a first column disposed between an adjacent second column and an adjacent third column, and selecting a header name for the first column further comprises:
identifying a first set of features based on the contents of the first column;
determining the first set of features is appropriate by reference to the contents of the second column and the third column; and
classifying, using the second CRF model and based on the first set of features, the first column as falling under a first class that is selected from a set of previously defined classes, the header name being associated with the first class.

### Claim 9

A system for tabular or list-based data extraction from document images, the system comprising:
a processor;
machine-readable media including instructions which, when executed by the processor, cause the processor to:
receive, at a server and from a first data source, a first document including a first page;
perform a column-wise pixel analysis of the first page, thereby determining that the first page includes a first table;
perform column segmentation based on signal analysis of column-wise mean pixel values of the first page, thereby identifying a set of columns;
perform row segmentation using optical character recognition (OCR)-generated bounding boxes, thereby identifying a set of rows;
select which rows of the set of rows belong to the first table using a first Conditional Random Fields (CRF) model, thereby localizing the first table on the first page;
select, for each column in the set of columns, a header name from a pre-defined set of header names, the selection being based on a classification performed by a second CRF model that evaluates at least the entire contents of that column;
map each item of data extracted from a cell in the first table to a field using semantic data understanding; and
generate a first digital table representing data extracted from the first table for presentation in a user interface.

### Claim 10 (depends on 9)

The system of claim 9, wherein the instructions further cause the processor to:
perform image binarization of the first page to produce a binarized page, where the binarized page is at a first orientation;
select a first range of rotation including a first number of angles extending in a sequence from a first angle to a second angle, the angles in the sequence being evenly spaced apart by a first increment;
incrementally rotate the binarized page to align with each angle in the first range of rotation such that the binarized page rotates step-wise from the first orientation until oriented at a second orientation associated with the second angle;
calculate, for each incremental rotation of the binarized page through the first range of rotation, a set of row-wise mean pixel values of the binarized page, thereby producing a plurality of row-wise mean pixel value sets, the plurality of row-wise mean pixel value sets including a first set of row-wise mean pixel values for the binarized page while at a third orientation associated with a third angle of the first range of rotation;
generate, for each orientation of the binarized page resulting from the incremental rotation through the first range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a first signal representing the first set of row-wise mean pixel values;
calculate, for each signal in the plurality of signals, a variance level of that signal;
determine the first signal has the highest variance level; and
select, for further processing, the binarized image in the third orientation based on the first signal having the highest variance level.

### Claim 11 (depends on 10)

The system of claim 10, wherein the instructions further cause the processor to:
select a second range of rotation including a second number of angles extending in a sequence from a fourth angle to a fifth angle, the angles in the sequence being evenly spaced apart by a second increment that is smaller than the first increment;
incrementally rotate the binarized page to align with each angle in the second range of rotation;
calculate, for each incremental rotation of the binarized page through the second range of rotation, a set of row-wise mean pixel values of the binarized page;
generate, for each orientation of the binarized page resulting from the incremental rotation through the second range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a second signal for the binarized page in a fourth orientation; and
determine the binarized image in the fourth orientation is less skewed than the binarized image in the third orientation if a variance level for the second signal is greater than the variance level for the first signal.

### Claim 12 (depends on 10)

The system of claim 10, wherein the second range of rotation includes the third angle.

### Claim 13 (depends on 10)

The system of claim 10, wherein the instructions further cause the processor to employ a minmax-concave total variation denoising technique to smooth the first signal.

### Claim 14 (depends on 9)

The system of claim 9, wherein the instructions further cause the processor to classify each row into one of three classes comprising a header row, a table row, and an other row, and a position of the first table on the first page is determined based on the classification of each row into one of the three classes.

### Claim 15 (depends on 9)

The system of claim 9, wherein the set of columns includes a first column and a directly adjacent second column, and the instructions further cause the processor to:
determine, using semantic data understanding, that first data in the first column and second data in the second column have a higher likelihood of representing meaningful content when evaluated as a single data entry; and
merge the first column and the second column into a single, third column.

### Claim 16 (depends on 9)

The system of claim 9, wherein the set of columns includes a first column disposed between an adjacent second column and an adjacent third column, and the instructions further cause the processor to:
identify a first set of features based on the contents of the first column,
determine whether the first set of features is appropriate by reference to the contents of the second column and the third column; and
classify, using the second CRF model and based on the first set of features, the first column as falling under a first class that is selected from a set of previously defined classes, the header name being associated with the first class.

### Claim 17

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to:
receive, at a server and from a first data source, a first document including a first page;
perform a column-wise pixel analysis of the first page, thereby determining that the first page includes a first table;
perform column segmentation based on signal analysis of column-wise mean pixel values of the first page, thereby identifying a set of columns;
perform row segmentation using optical character recognition (OCR)-generated bounding boxes, thereby identifying a set of rows;
select which rows of the set of rows belong to the first table using a first Conditional Random Fields (CRF) model, thereby localizing the first table on the first page;
select, for each column in the set of columns, a header name from a pre-defined set of header names, the selection being based on a classification performed by a second CRF model that evaluates at least the entire contents of that column;
map each item of data extracted from a cell in the first table to a field using semantic data understanding; and
generate a first digital table representing data extracted from the first table for presentation in a user interface.

### Claim 18 (depends on 17)

The system of claim 17, wherein the instructions further cause the one or more computers to:
perform image binarization of the first page to produce a binarized page, where the binarized page is at a first orientation;
select a first range of rotation including a first number of angles extending in a sequence from a first angle to a second angle, the angles in the sequence being evenly spaced apart by a first increment;
incrementally rotate the binarized page to align with each angle in the first range of rotation such that the binarized page rotates step-wise from the first orientation until oriented at a second orientation associated with the second angle;
calculate, for each incremental rotation of the binarized page through the first range of rotation, a set of row-wise mean pixel values of the binarized page, thereby producing a plurality of row-wise mean pixel value sets, the plurality of row-wise mean pixel value sets including a first set of row-wise mean pixel values for the binarized page while at a third orientation associated with a third angle of the first range of rotation;
generate, for each orientation of the binarized page resulting from the incremental rotation through the first range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a first signal representing the first set of row-wise mean pixel values;
calculate, for each signal in the plurality of signals, a variance level of that signal;
determine the first signal has the highest variance level; and
select, for further processing, the binarized image in the third orientation based on the first signal having the highest variance level.

### Claim 19 (depends on 18)

The system of claim 18, wherein the instructions further cause the processor to:
select a second range of rotation including a second number of angles extending in a sequence from a fourth angle to a fifth angle, the angles in the sequence being evenly spaced apart by a second increment that is smaller than the first increment;
incrementally rotate the binarized page to align with each angle in the second range of rotation;
calculate, for each incremental rotation of the binarized page through the second range of rotation, a set of row-wise mean pixel values of the binarized page;
generate, for each orientation of the binarized page resulting from the incremental rotation through the second range of rotation, a signal representing the set of row-wise mean pixel values of the binarized page while in that orientation, thereby producing a plurality of signals that includes a second signal for the binarized page in a fourth orientation; and
determine the binarized image in the fourth orientation is less skewed than the binarized image in the third orientation if a variance level for the second signal is greater than the variance level for the first signal.

### Claim 20 (depends on 17)

The system of claim 17, wherein the set of columns includes a first column disposed between an adjacent second column and an adjacent third column, and the instructions further cause the processor to:
identify a first set of features based on the contents of the first column,
determine whether the first set of features is appropriate by reference to the contents of the second column and the third column; and
classify, using the second CRF model and based on the first set of features, the first column as falling under a first class that is selected from a set of previously defined classes, the header name being associated with the first class.