# US 12,217,497 B2 · Extracting key information from document using trained machine-learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,497 B2 - Document ID: US12217497B2 - Status: Granted - Kind: B2 - Application number: 17/888,300 - Filed: 2022-08-15 - Published: 2023-03-30 - Granted: 2025-02-04 - Earliest claimed: 2021-09-28 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Yakupitiyage Don Thanuja Samodhye Dharmasiri; Xu Zhong; Ahmed Ataallah Ataallah Abobakr; Hongtao Yang; Budhaditya Saha; Shaoke Xu; Shashi Prasad Suravarapu; Mark Edward Johnson; Thanh Long Duong - Web page: https://patentgrep.com/patent/US12217497B2 ## CPC - G06V 10/82: using neural networks - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 30/153: {using recognition of characters or words} - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques for extracting key information from a document using machine-learning models in a chatbot system is disclosed herein. In one particular aspect, a method is provided that includes receiving a set of data, which includes key fields, within a document at a data processing system that includes a table detection module, a key information extraction module, and a table extraction module. Text information and corresponding location data are extracted via optical character recognition. The table detection module detects whether one or more tables are present in the document and, if applicable, a location of each of the tables. The key information extraction module extracts text from the key fields. The table extraction module extracts each of the tables based on input from the optical character recognition and the table detection module. Extraction results include the text from the key fields and each of the tables can be output. ## Claims ### Claim 1 A method comprising: receiving, at a data processing system, a set of data within a document, the set of data including one or more key fields, wherein the data processing system comprises a table detection module, a key information extraction module, and a table extraction module; extracting, by the data processing system, text information and location data associated with the text information by performing an optical character recognition operation, the text information and the location data from the document; detecting, by the table detection module, whether one or more tables are present in the document, wherein the table detection module is a trained object detection neural network trained using document images for outputting a plurality of pixel coordinates of one or more tables in the document images; for each of the one or more tables determined to be within the document, detecting, by the table detection module, a location of each of the one or more tables included in the document; extracting, by the key information extraction module and based on the text information and the location data, text from the one or more key fields; extracting, by the table extraction module, each of the one or more tables determined to be within the document based on: (i) text information and location data associated with the text information, and (ii) the location of each of the one or more tables; and outputting, by the data processing system, extraction results for the document, the extraction results including the text from the one or more key fields, and the one or more tables. ### Claim 2 (depends on 1) The method of claim 1, wherein the key information extraction module is a first trained neural network and the table extraction module is a third trained neural network, and wherein the first trained neural network, the trained object detection neural network, and the third trained neural network are trained separately using different training processes. ### Claim 3 (depends on 1) The method of claim 1, wherein the key information extraction module is a trained neural network that is trained via cross-entropy loss, wherein data used in the cross- entropy loss includes synthetic data generated using real data as a template of the synthetic data, and wherein the cross-entropy loss includes a location graph having a graph convolutional network applied to the location graph. ### Claim 4 (depends on 3) The method of claim 3, wherein the location graph is a first location graph, wherein the cross-entropy loss includes a second location graph, wherein the first location graph represents a relative vertical location of the synthetic data, and wherein the second location graph represents a relative horizontal location of the synthetic data. ### Claim 5 (depends on 1) The method of claim 1, wherein the table extraction module is a trained neural network that is trained by: receiving a bounding box of a table of a training document and a plurality of bounding boxes corresponding to a plurality of cells of the table; extracting tokens within the table; and labeling table coordinates of the tokens, the table coordinates including a starting row, a starting column, a row span, and a column span. ### Claim 6 (depends on 1) The method of claim 1, wherein extracting, by the table extraction module, each of the one or more tables determined to be within the document includes: extracting a plurality of tokens from the one or more tables, the plurality of tokens corresponding to a plurality of cells of the one or more tables; determining, based on the plurality of tokens, table coordinates of the plurality of tokens, the table coordinates including a starting row, a starting column, a row span, and a column span; and using the table coordinates and the text information and location data associated with the text information to extract each of the one or more tables. ### Claim 7 A system comprising: one or more processors; and a non-transitory computer-readable medium coupled to the one or more processors, the non-transitory computer-readable medium storing instructions executable by the one or more processors to cause the one or more processors to perform operations comprising: receiving a set of data within a document, the set of data including one or more key fields, wherein the data processing system comprises a table detection module, a key information extraction module, and a table extraction module; extracting text information and location data associated with the text information by performing an optical character recognition operation, the text information and the location data from the document; detecting, by the table detection module, whether one or more tables are present in the document, wherein the table detection module is a trained object detection neural network trainable using document images for outputting a plurality of pixel coordinates of one or more tables in the document images; for each of the one or more tables determined to be within the document, detecting, by the table detection module, a location of each of the one or more tables included in the document; extracting, by the key information extraction module and based on the text information and the location data, text from the one or more key fields; extracting, by the table extraction module, each of the one or more tables determined to be within the document based on: (i) text information and location data associated with the text information, and (ii) the location of each of the one or more tables; and outputting extraction results for the document, the extraction results including the text from the one or more key fields, and the one or more tables. ### Claim 8 (depends on 7) The system of claim 7, wherein the key information extraction module is a first trained neural network and the table extraction module is a third trained neural network, and wherein the first trained neural network, the trained object detection neural network, and the third trained neural network are trained separately using different training processes. ### Claim 9 (depends on 7) The system of claim 7, wherein the key information extraction module is a trained neural network that is trainable via cross-entropy loss, wherein data used in the cross-entropy loss includes synthetic data generateable using real data as a template of the synthetic data, and wherein the cross-entropy loss includes a location graph having a graph convolutional network applicable to the location graph. ### Claim 10 (depends on 9) The system of claim 9, wherein the location graph is a first location graph, wherein the cross-entropy loss includes a second location graph, wherein the first location graph represents a relative vertical location of the synthetic data, and wherein the second location graph represents a relative horizontal location of the synthetic data. ### Claim 11 (depends on 7) The system of claim 7, wherein the table extraction module is a trained neural network that is trainable by: receiving a bounding box of a table of a training document and a plurality of bounding boxes corresponding to a plurality of cells of the table; extracting tokens within the table; and labeling table coordinates of the tokens, the table coordinates including a starting row, a starting column, a row span, and a column span. ### Claim 12 (depends on 7) The system of claim 7, wherein the operation of extracting, by the table extraction module, each of the one or more tables determined to be within the document includes: extracting a plurality of tokens from the one or more tables, the plurality of tokens corresponding to a plurality of cells of the one or more tables; determining, based on the plurality of tokens, table coordinates of the plurality of tokens, the table coordinates including a starting row, a starting column, a row span, and a column span; and using the table coordinates and the text information and location data associated with the text information to extract each of the one or more tables. ### Claim 13 A non-transitory computer-readable medium storing instructions executable by one or more processors for causing the one or more processors to perform operations comprising: receiving a set of data within a document, the set of data including one or more key fields, wherein the data processing system comprises a table detection module, a key information extraction module, and a table extraction module; extracting text information and location data associated with the text information by performing an optical character recognition operation, the text information and the location data from the document; detecting, by the table detection module, whether one or more tables are present in the document, wherein the table detection module is a trained object detection neural network trainable using document images for outputting a plurality of pixel coordinates of one or more tables in the document images; for each of the one or more tables determined to be within the document, detecting, by the table detection module, a location of each of the one or more tables included in the document; extracting, by the key information extraction module and based on the text information and the location data, text from the one or more key fields; extracting, by the table extraction module, each of the one or more tables determined to be within the document based on: (i) text information and location data associated with the text information, and (ii) the location of each of the one or more tables; and outputting extraction results for the document, the extraction results including the text from the one or more key fields, and the one or more tables. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the key information extraction module is a first trained neural network the table extraction module is a third trained neural network, and wherein the first trained neural network, the trained object detection neural network, and the third trained neural network are trained separately using different training processes. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the key information extraction module is a trained neural network that is trainable via cross-entropy loss, wherein data used in the cross-entropy loss includes synthetic data generateable using real data as a template of the synthetic data, wherein the cross-entropy loss includes a location graph having a graph convolutional network applicable to the location graph, wherein the location graph is a first location graph, wherein the cross-entropy loss includes a second location graph, wherein the first location graph represents a relative vertical location of the synthetic data, and wherein the second location graph represents a relative horizontal location of the synthetic data. ### Claim 16 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the table extraction module is a trained neural network that is trainable by: receiving a bounding box of a table of a training document and a plurality of bounding boxes corresponding to a plurality of cells of the table; extracting tokens within the table; and labeling table coordinates of the tokens, the table coordinates including a starting row, a starting column, a row span, and a column span. ### Claim 17 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the operation of extracting, by the table extraction module, each of the one or more tables determined to be within the document includes: extracting a plurality of tokens from the one or more tables, the plurality of tokens corresponding to a plurality of cells of the one or more tables; determining, based on the plurality of tokens, table coordinates of the plurality of tokens, the table coordinates including a starting row, a starting column, a row span, and a column span; and using the table coordinates and the text information and location data associated with the text information to extract each of the one or more tables.