# US 2021/0240976 A1 · Machine learning based end-to-end extraction of tables from electronic documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240976 A1 - Document ID: US20210240976A1 - Status: Published application - Kind: A1 - Application number: 16/781,195 - Filed: 2020-02-04 - Published: 2021-08-05 - Earliest claimed: 2020-02-04 - Assignees: Eygs LLP - Applicants: Eygs LLP - Inventors: Sunil Reddy TIYYAGURA; Amani KONGARA - Web page: https://patentgrep.com/patent/US20210240976A1 ## CPC - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - G06F 40/295: Named entity recognition - G06V 30/10: Character recognition - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In some embodiments, a method includes identifying a set of word bounding boxes in a first electronic document, and identifying locations of horizontal white space between two adjacent rows from a set of rows in a table. The method includes determining, using a Natural Language Processing algorithm, an entity name from a set of entity names for each table cell from a set of table cells in the table. The method includes determining, using a machine learning algorithm a class from a set of classes for each row from the set of rows. The method includes extracting a set of table cell values associated with the set of table cells, and generating a second electronic document including the set of table cell values arranged in the set of rows and the set of columns such that the set of words in the table are computer-readable in the second electronic document. ## Claims ### Claim 1 A method, comprising: identifying a plurality of word bounding boxes in a first electronic document, each word bounding box from the plurality of word bounding boxes associated with a word from a plurality of words in the first electronic document, the first electronic document including a table having a set of table cells positioned in a set of rows and a set of columns; identifying, based on coordinates of the plurality of word bounding boxes, locations of horizontal white space between two adjacent rows from the set of rows; determining, using a Natural Language Processing algorithm, an entity name from a plurality of entity names for each table cell from the set of table cells; determining, using a machine learning algorithm and based on (1) the locations of horizontal white space and (2) the plurality of entity names, a class from a set of classes for each row from the set of rows in the table; extracting a set of table cell values associated with the set of table cells based on, for each row from the set of rows, the class from the set of classes for that row; and generating a second electronic document including the set of table cell values arranged in the set of rows and the set of columns and based on (1) the locations of horizontal white space, (2) locations of vertical white space, such that the plurality of words in the table are computer-readable in the second electronic document. ### Claim 2 (depends on 1) The method of claim 1, wherein the first electronic document is a scanned image. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving the first electronic document from an optical character recognition program. ### Claim 4 (depends on 1) The method of claim 1, wherein: the first electronic document includes a first page and a second page; the method further includes generating a third electronic document by appending a vertical coordinate of the second page with a vertical coordinate of the first page, the third electronic document having one page including the plurality of words from the first page and the second page. ### Claim 5 (depends on 1) The method of claim 1, wherein the first electronic document includes a plurality of tables including the table. ### Claim 6 (depends on 1) The method of claim 1, wherein the set of classes includes at least one of Header, Row, Partial Row, or Table End. ### Claim 7 (depends on 1) The method of claim 1, wherein the second electronic document is in a format of Comma-Separated Values (CSV), Excel, or JavaScript Object Notation (JSON). ### Claim 8 An apparatus, comprising: a memory; and a processor operatively coupled to the memory, the processor configured to: receive a first electronic document and information on a character bounding box from a plurality of character bounding boxes and for each character from a plurality of characters in the first electronic document, identify, based on the plurality of character bounding boxes, a plurality of word bounding boxes, each word bounding box from the plurality of word bounding boxes associated with a word from a plurality of words in the first electronic document, determine, based on locations of the plurality of word bounding boxes, white space in a horizontal dimension of the first electronic document to identify a plurality of rows of a table included in the first electronic document, each row from the plurality of rows (1) including a set of words from the plurality of words and (2) being divided into a plurality of cells separated by the white space in the horizontal dimension, determine, using a machine learning algorithm, a class from a set of classes for each row from the plurality of rows in the table, extract, for each row from a subset of rows from the plurality of rows and based on the class from the set of classes for that row, a first set of word values associated with a set of words included in that row, generate a second electronic document based on (1) the first set of word values in each row from the subset of rows, (2) a second set of word values associated with a set of words included in remaining rows from the plurality of rows, (3) the plurality of rows of the table, (4) the plurality of columns of the table, (5) the white space in the horizontal dimension, and (6) white space in a vertical dimension such that the plurality of words in the table are computer-readable in the second electronic document, the remaining rows from the plurality of rows being different from the subset of rows from the plurality of rows. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the first electronic document is a scanned image. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the processor is configured to receive the first electronic document from an optical character recognition program. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein: the first electronic document includes a first page and a second page; the processor is configured to generate a third electronic document by appending a vertical coordinate of the second page with a vertical coordinate of the first page, the third electronic document having one page including the plurality of characters from the first page and the second page. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the first electronic document includes a plurality of tables including the table. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the set of classes includes at least one of Header, Row, Partial Row, or Table End. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the second electronic document is in a format of Comma-Separated Values (CSV), Excel, or JavaScript Object Notation (JSON). ### Claim 15 (depends on 8) The apparatus of claim 8, wherein: the set of classes includes at least one of Header, Row, Partial Row, or Table End; each row from the subset of rows has a class of Row; each row from the remaining rows has a class of Partial Row. ### Claim 16 (depends on 8) The apparatus of claim 8, wherein: the processor is configured to determine, using a Natural Language Processing algorithm, an entity name from a plurality of entity names for each cell from the plurality of cells in each row from the plurality of rows, the processor is configured to determine the set of classes by using the plurality of entity names. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the plurality of entity names includes Date, Person, Location, Organization, Number, Money, Cardinal, or No Entity. ### Claim 18 (depends on 8) The apparatus of claim 8, wherein the machine learning algorithm includes a set of weight values indicating probabilities of transition from one class from the set of classes to another class from the set of classes. ### Claim 19 (depends on 8) The apparatus of claim 8, wherein the table spans across multiple pages in the first electronic document. ### Claim 20 (depends on 8) The apparatus of claim 8, wherein the processor is configured to determine, based on (1) the locations of the plurality of word bounding boxes and (2) the set of classes for the plurality of rows, the white space in the vertical dimension of the first electronic document. ### Claim 21 A processor-readable non-transitory medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to: receive a first electronic document and locations of a plurality of character bounding boxes for each character from a plurality of characters in the first electronic document; identify, based on the locations of the plurality of character bounding boxes, a plurality of word bounding boxes, each word bounding box from the plurality of word bounding boxes associated with a word from a plurality of words in the first electronic document, the first electronic document including a table having a set of table cells positioned in a set of rows and a set of columns; identify, based on coordinates of the plurality of word bounding boxes, locations of horizontal white space between two adjacent rows from the set of rows; determine, using a machine learning algorithm and based on the locations of horizontal white space, a class from a set of classes for each row from the set of rows in the table; extract a set of table cell values associated with the set of table cells; and generate a second electronic document including the set of table cell values arranged in the set of rows and the set of columns and based on (1) the locations of horizontal white space, (2) locations of vertical white space, such that the plurality of words in the table are computer-readable in the second electronic document.