# US 2019/0317985 A1 · Generating a structured document based on a machine readable document and artificial intelligence-generated annotations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0317985 A1 - Document ID: US20190317985A1 - Status: Published application - Kind: A1 - Application number: 15/950,515 - Filed: 2018-04-11 - Published: 2019-10-17 - Earliest claimed: 2018-04-11 - Assignees: Accenture Global Solutions Limited - Applicants: Accenture Global Solutions Limited - Inventors: Gino Andre Di Paolo; Bogdan Eugen Sacaleanu; Urvesh Bhowan; Robert Drysdale - Web page: https://patentgrep.com/patent/US20190317985A1 ## CPC - G06V 30/40: Document-oriented image-based pattern recognition - G06F 16/334: Query execution (filtering based on additional data G06F16/335) - G06F 40/169: Annotation, e.g. comment data or footnotes - G06N 5/02: Knowledge representation; Symbolic representation - G06V 20/20: in augmented reality scenes - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A device obtains a machine readable document, and extracts plain text information from the machine readable document. The device extracts visual structural information from the machine readable document, and processes the plain text information with an artificial intelligence model to generate artificial intelligence-generated annotations. The device aligns the artificial intelligence-generated annotations and the visual structural information to generate a structured document, and provides the structured document for display. ## Claims ### Claim 1 A device, comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, to: obtain a machine readable document; extract plain text information from the machine readable document; create a first document layer that includes the plain text information; extract visual structural information from the machine readable document; create a second document layer that includes the visual structural information; process the plain text information with an artificial intelligence model to generate artificial intelligence-generated annotations for the first document layer; align and combine the artificial intelligence-generated annotations of the first document layer and the visual structural information of the second document layer to generate a structured document, the artificial intelligence-generated annotations being mapped with coordinates in the visual structural information, and the visual structural information and the artificial intelligence-generated annotations being combined, to generate the structured document, based on mapping the artificial intelligence-generated annotations with the coordinates; provide the structured document for display; receive an additional annotation after providing the structured document for display; process the additional annotation with the artificial intelligence model to identify the additional annotation in the first document layer; align and combine the first document layer, with the artificial intelligence-generated annotations, the second document layer, and the additional annotation to generate a modified structured document; and update the artificial intelligence model based on the additional annotation. ### Claim 2 (depends on 1) The device of claim 1, wherein the one or more processors, when obtaining the machine readable documents, are to: receive a non-machine readable document; and process the non-machine readable document to generate the machine readable document. ### Claim 3 (depends on 1) The device of claim 1, wherein the one or more processors, when processing the plain text information with the artificial intelligence model, are to: search for matching particular information in the plain text information; and generate the artificial intelligence-generated annotations based on the matching particular information. ### Claim 4 (canceled) ### Claim 5 (depends on 1) The device of claim 1, wherein the artificial intelligence model includes one or more of: a named-entity recognition (NER) model, a conditional random field (CRF) model, a naïve string search model, a Rabin-Karp string search model, a Knuth-Morris-Pratt model, a Boyer-Moore string search model, or a two-way string-matching model. ### Claim 6 (canceled) ### Claim 7 (depends on 1) The device of claim 1, wherein the one or more processors, when extracting the plain text information from the machine readable document, are to: utilize a full extraction method or an incremental extraction method to extract the plain text information from the machine readable document. ### Claim 8 A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors, cause the one or more processors to: receive a non-machine readable document; process the non-machine readable document to generate a machine readable document; extract plain text information from the machine readable document; create a first document layer that includes the plain text information; extract visual structural information from the machine readable document; create a second document layer that includes the visual structural information; process the plain text information with an artificial intelligence model to identify artificial intelligence-generated annotations for the first document layer; align and combine the first document layer, with the artificial intelligence-generated annotations, and the second document layer to generate a structured document, the artificial intelligence-generated annotations being mapped with coordinates in the visual structural information, and the visual structural information and the artificial intelligence-generated annotations being combined, to generate the structured document, based on mapping the artificial intelligence-generated annotations with the coordinates; provide the structured document for display; receive an additional annotation after providing the structured document for display; process the additional annotation with the artificial intelligence model to identify the additional annotation in the first document layer; align and combine the first document layer, with the artificial intelligence-generated annotations, the second document layer, and the additional annotation to generate a modified structured document; and update the artificial intelligence model based upon the additional annotation. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the one or more instructions that cause the one or more processors to process the non-machine readable document, comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to: process the non-machine readable document, using one or more techniques, to generate the machine readable document, wherein the one or more techniques include: optical character recognition (OCR), a natural language processing technique, a computational linguistics technique, or a text analysis technique. ### Claim 10 (canceled) ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the one or more instructions that cause the one or more processors to process the plain text information with the artificial intelligence model, comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to: match particular information in the plain text information; and identify the artificial intelligence-generated annotations, in the plain text information, based on the matching particular information. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the artificial intelligence model includes an artificial intelligence model that: analyzes text, identifies particular information in the text based on analyzing the text, and provides annotations to the particular information in the text. ### Claim 13 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the one or more instructions that cause the one or more processors to extract the visual structural information from the machine readable document, comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to: utilize a full extraction method or an incremental extraction method to extract the visual structural information from the machine readable document. ### Claim 14 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the structured document includes: the first document layer with the artificial intelligence-generated annotations, and the second document layer with the visual structural information, wherein the first document layer, with the artificial intelligence-generated annotations, and the second document layer are combined into a single document. ### Claim 15 A method, comprising: receiving, by a device, a non-machine readable document; processing, by the device, the non-machine readable document to generate a machine readable document; extracting, by the device, plain text information from the machine readable document; creating, by the device, a first document layer that includes the plain text information; extracting, by the device, visual structural information from the machine readable document; creating, by the device, a second document layer that includes the visual structural information; processing, by the device, the plain text information with an artificial intelligence model to identify artificial intelligence-generated annotations in the plain text information for the first document layer; mapping, by the device, the artificial intelligence-generated annotations with coordinates of the visual structural information; aligning and combining, by the device, the visual structural information of the second document layer and the artificial intelligence-generated annotations of the first document layer, to generate a structured document, based on mapping the artificial intelligence-generated annotations with the coordinates; providing, by the device, the structured document for display; receiving, by the device, an additional annotation after providing the structured document for display; processing, by the device, the additional annotation with the artificial intelligence model to identify the additional annotation in the first document layer; aligning and combining, by the device, the first document layer, with the artificial intelligence-generated annotations, the second document layer, and the additional annotation to generate a modified structured document; and updating, by the device, the artificial intelligence model based upon the additional annotation. ### Claim 16 (depends on 15) The method of claim 15, wherein processing the non-machine readable document includes: processing the non-machine readable document, using optical character recognition (OCR), to convert the non-machine readable document into the machine readable document. ### Claim 17 (depends on 15) The method of claim 15, wherein processing the plain text information with the artificial intelligence model includes: searching for matching particular information in the plain text information; and generating the artificial intelligence-generated annotations based on the matching particular information. ### Claim 18 (depends on 15) The method of claim 15, wherein the first document layer, with the artificial intelligence-generated annotations, and the second document layer are combined into a single document. ### Claim 19 (canceled) ### Claim 20 (depends on 15) The method of claim 15, wherein the artificial intelligence model includes an artificial intelligence model that analyzes text, identifies particular information in the text based on analyzing the text, and provides annotations to the particular information in the text. ### Claim 21 (depends on 1) The device of claim 1, wherein the one or more processors are further to: receive a non-machine readable document; and process the non-machine readable document to generate the machine readable document, wherein the non-machine readable document includes at least one of: one or more scanned portable document format (PDF) documents, one or more images, one or more audio files, or one or more video files. ### Claim 22 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the non-machine readable document includes at least one of: one or more scanned portable document format (PDF) documents, one or more images, one or more audio files, or one or more video files. ### Claim 23 (depends on 1) The device of claim 1, wherein the one or more processors, when obtaining the machine readable documents, are to: receive a non-machine readable document; and utilize speech recognition to generate the machine readable document. ### Claim 24 (depends on 15) The method of claim 15, wherein processing the non-machine readable document to generate the machine readable document comprises: utilizing speech recognition to generate the machine readable document.