# US 12,217,524 B2 · Systems and methods for automated end-to-end text extraction of electronic documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,524 B2 - Document ID: US12217524B2 - Status: Granted - Kind: B2 - Application number: 17/850,618 - Filed: 2022-06-27 - Published: 2023-12-28 - Granted: 2025-02-04 - Earliest claimed: 2022-06-27 - Assignees: Fmr LLC - Applicants: Fmr LLC - Inventors: Keerthan Ramnath; Punitha Chandrasekar; Hui Su; Shyam Subramanian; Rachna Saxena; Mohamed Mahdi Alouane; Vinay Iyengar - Web page: https://patentgrep.com/patent/US12217524B2 ## CPC - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - G06F 40/232: Orthographic correction, e.g. spell checking or vowelisation - G06F 40/263: Language identification - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06V 30/10: Character recognition - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods for extracting data from electronic documents using optical character recognition (OCR) and non-OCR based text extraction. A server computing device initiates non-OCR based text extraction for each page of an electronic document. The server calculates a document text coverage percentage corresponding to the non-OCR based text extraction for the whole document and, in response to determining that the document text coverage percentage is below a first threshold, initiates OCR for the document. The server calculates a page text coverage percentage corresponding to the non-OCR based text extraction for one or more pages of the electronic document and, in response to determining that the page text coverage percentage is below a second threshold, initiates OCR for the pages. The server combines first text extracted from the electronic document using non-OCR based text extraction and second text extracted from the electronic document using OCR. ## Claims ### Claim 1 A computerized method for extracting data from electronic documents using optical character recognition (OCR) and non-OCR based text extraction, the method comprising: initiating, by a server computing device, non-OCR based text extraction for each of a plurality of pages of an electronic document; calculating, by the server computing device, a document text coverage percentage corresponding to the non-OCR based text extraction for the electronic document as a whole; in response to determining that the document text coverage percentage for the electronic document as a whole is below a first threshold, initiating, by the server computing device, OCR for the electronic document as a whole; calculating, by the server computing device, a page text coverage percentage corresponding to the non-OCR based text extraction for one or more pages of the electronic document; in response to determining that the page text coverage percentage for one or more pages of the electronic document is below a second threshold, initiating, by the server computing device, OCR for the one or more pages of the electronic document; and combining, by the server computing device, first text extracted from the electronic document using non-OCR based text extraction and second text extracted from the electronic document using OCR. ### Claim 2 (depends on 1) The computerized method of claim 1, wherein the server computing device is further configured to: generate a data structure comprising the combined first text and second text extracted from the electronic document; and store the data structure in a database. ### Claim 3 (depends on 1) The computerized method of claim 1, wherein the server computing device determines, for each of the plurality of pages of the electronic document, whether the page of the plurality of pages of the electronic document comprises a non-searchable image. ### Claim 4 (depends on 3) The computerized method of claim 3, wherein the server computing device, in response to determining that the page of the plurality of pages of the electronic document comprises the non-searchable image, initiates optical character recognition for the page of the plurality of pages of the electronic document. ### Claim 5 (depends on 1) The computerized method of claim 1, wherein the server computing device calculates the accuracy of the non-OCR based text extraction by calculating a percentage of extracted words that are English words. ### Claim 6 (depends on 1) The computerized method of claim 1, wherein the server computing device removes one or more watermarks from the electronic document prior to initiating optical character recognition. ### Claim 7 (depends on 1) The computerized method of claim 1, wherein the server computing device corrects the combined first text and second text extracted from the electronic document by replacing misspelled words. ### Claim 8 (depends on 1) The computerized method of claim 1, wherein the first threshold comprises 90% of the plurality of pages of the electronic document. ### Claim 9 (depends on 1) The computerized method of claim 1, wherein the second threshold comprises an image coverage of 50%. ### Claim 10 (depends on 1) The computerized method of claim 1, wherein the third threshold comprises an accuracy of 85%. ### Claim 11 A system for extracting data from electronic documents using non-optical character recognition (OCR) based text extraction and OCR, the system comprising a server computing device communicatively coupled to a database over a network, the server computing device having a memory for storing computer-executable instructions and a processor that executes the computer-executable instructions to: initiate non-OCR based text extraction for each of a plurality of pages of an electronic document; determine an amount of pages of the plurality of pages comprising at least one image; receive a runtime exception during non-OCR based text extraction or determine that the amount of pages exceeds a first threshold; initiate optical character recognition for each of the plurality of pages of the electronic document; determine an image coverage percentage for each of the plurality of pages of the electronic document; determine that the image coverage percentage exceeds a second threshold for a page of the plurality of pages of the electronic document; initiate optical character recognition for the page of the plurality of pages of the electronic document; calculate an accuracy of the non-OCR based text extraction performed for each of the plurality of pages of the electronic document; determine that the accuracy of the non-OCR based text extraction is less than a third threshold for a page of the plurality of pages of the electronic document; initiate optical character recognition for the page of the plurality of pages of the electronic document; and combine first text extracted from the electronic document using non-OCR based text extraction and second text extracted from the electronic document using optical character recognition. ### Claim 12 (depends on 11) The system of claim 11, wherein the server computing device determines, for each of the plurality of pages of the electronic document, whether the page of the plurality of pages of the electronic document comprises a non-searchable image. ### Claim 13 (depends on 12) The system of claim 12, wherein the server computing device, in response to determining that the page of the plurality of pages of the electronic document comprises the non-searchable image, initiates optical character recognition for the page of the plurality of pages of the electronic document. ### Claim 14 (depends on 11) The system of claim 11, wherein the server computing device calculates the accuracy of the non-OCR based text extraction by calculating a percentage of extracted words that are English words. ### Claim 15 (depends on 11) The system of claim 11, wherein the server computing device removes one or more watermarks from the electronic document prior to initiating optical character recognition. ### Claim 16 (depends on 11) The system of claim 11, wherein the server computing device corrects the combined first text and second text extracted from the electronic document by replacing misspelled words. ### Claim 17 (depends on 11) The system of claim 11, wherein the first threshold comprises 90% of the plurality of pages of the electronic document. ### Claim 18 (depends on 11) The system of claim 11, wherein the second threshold comprises an image coverage of 50%. ### Claim 19 (depends on 11) The system of claim 11, wherein the third threshold comprises an accuracy of 85%. ### Claim 20 A computerized method for extracting data from electronic documents using non-optical character recognition (OCR) based text extraction and OCR, the method comprising: initiating, by a server computing device, non-OCR based text extraction for each of a plurality of pages of an electronic document; determining, by the server computing device, an amount of pages of the plurality of pages comprising at least one image; receiving, by the server computing device, a runtime exception during non-OCR based text extraction or determining, by the server computing device, that the amount of pages exceeds a first threshold; initiating, by the server computing device, optical character recognition for each of the plurality of pages of the electronic document; determining, by the server computing device, an image coverage percentage for each of the plurality of pages of the electronic document; determining, by the server computing device, that the image coverage percentage exceeds a second threshold for a page of the plurality of pages of the electronic document; initiating, by the server computing device, optical character recognition for the page of the plurality of pages of the electronic document; calculating, by the server computing device, an accuracy of the non-OCR based text extraction performed for each of the plurality of pages of the electronic document; determining, by the server computing device, that the accuracy of the non-OCR based text extraction is less than a third threshold for a page of the plurality of pages of the electronic document; initiating, by the server computing device, optical character recognition for the page of the plurality of pages of the electronic document; and combining, by the server computing device, first text extracted from the electronic document using non-OCR based text extraction and second text extracted from the electronic document using optical character recognition.