# US 12,008,028 B2 · Mining locations and other context information from construction documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,028 B2 - Document ID: US12008028B2 - Status: Granted - Kind: B2 - Application number: 16/174,757 - Filed: 2018-10-30 - Published: 2020-04-30 - Granted: 2024-06-11 - Earliest claimed: 2018-10-30 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Elham Khabiri; Pietro Mazzoleni - Web page: https://patentgrep.com/patent/US12008028B2 ## CPC - G06F 16/367: {Ontology} - G06F 16/387: using geographical or spatial information, e.g. location - G06F 40/30: Semantic analysis - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 5/046: Forward inferencing; Production systems - G06V 30/422: Technical drawings; Geographical maps - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of processing construction documentation includes receiving, as input data into a processor of a computer, a document directed to a specific construction project as input data. The processor extracts location and context information from the document. The location information defines locations relevant to the specific construction project and the context information comprises text extracted from image data of the document. ## Claims ### Claim 1 A method of processing construction documentation, the method comprising: using at least one industrial standard to develop an ontology and a taxonomy as defining a domain of construction documents to be used for processing a plurality of engineering documents directed to a specific construction project, the ontology comprising a set of concepts and categories in the domain that shows their properties and interrelationships and the taxonomy comprising a common vocabulary to be used for terms found in the plurality of engineering documents directed to the specific construction project; receiving an input area that describes where in sample engineering documents to search for location information; training a learning model, using a processor on a computer, to extract location information from the input area and context information including semantic information from the sample engineering documents, using the developed ontology and taxonomy such that the extracted context information is semantically related to said taxonomy; applying the trained learning model to the plurality of engineering documents directed to the specific construction project, comprising: receiving as input data the plurality of engineering documents that comprises a data-dump of one or more three-dimensional (3D) models of the specific construction project and corresponding architecture and an overall layout for the specific construction project; and providing the input data to the trained learning model that extracts location information by automatically inferring grid coordinates in the input data even when grid coordinates in an engineering document are obscured by other content in the engineering document, wherein inferring grid coordinates includes interpolating visible information from coordinates mined from drawings of other floors in the plurality of engineering documents; upon completion of applying the trained learning model, storing each engineering document in a database of construction documents directed to the specific construction project, each engineering document being stored as annotated with its extracted location information and context information; and providing an interface that provides access to the database of construction documents directed to the specific construction project. ### Claim 2 (depends on 1) The method of claim 1, the interface comprising a natural language processing (NLP) that permits a user to query the database of construction documents using a natural language. ### Claim 3 (depends on 2) The method of claim 2, as implemented as a cloud service. ### Claim 4 (depends on 1) The method of claim 1, further comprising: wherein the extracted location information defines physical locations relevant to the specific construction project; wherein the extracted context information comprises text extracted from image data of an engineering document; and wherein extracting the location information and the context information including running analytics on an engineering document at a page-level granularity and extraction from metadata of the engineering document. ### Claim 5 (depends on 4) The method according to claim 4, wherein the receiving as input data the plurality of engineering documents is from physical or virtual resources, the location information includes location hypothesis information, and extracting the context information includes analytics on the location hypothesis information that runs at different level of granularity. ### Claim 6 (depends on 4) The method of claim 4, wherein extracting the location information and the context information further comprising: using a location hypothesis extraction module to extract location information from engineering documents by analyzing file names, file metadata, and file content to mine the location information and provide a first output to a location semantic classifier; using a location semantic classifier to semantically correlate the first output from the location hypothesis extraction module with external data sources to provide a second output; and using location context mining to merge the second output of the location semantic classifier with external documents to provide a third output. ### Claim 7 (depends on 4) The method of claim 4, further comprising applying a natural language processing (NLP) to the extracted location information to semantically correlate the context information with different contexts related to the specific construction project. ### Claim 8 (depends on 4) The method of claim 4, further comprising applying an optical character recognition (OCR) processing for the extracting of the location information and the context information. ### Claim 9 (depends on 4) The method of claim 4, further comprising: annotating each engineering document with corresponding location information and context information extracted from the engineering document; and storing each annotated engineering document in the database of construction documents directed to the specific construction project. ### Claim 10 (depends on 4) The method of claim 4, as implemented on an analytics service platform. ### Claim 11 (depends on 1) The method of claim 1, further comprising: inferring physical locations and context information from the plurality of engineering documents where such information is not available either as metadata or directly accessible from a three-dimensional (3D) model of the specific construction project. ### Claim 12 (depends on 1) The method of claim 1, wherein the learning model is an end-to-end (E2E) analytic algorithm. ### Claim 13 A computer system comprising: one or more computer processors; one or more computer readable memories; and one or more computer readable storage media; program instructions, stored on at least one of the one or more computer readable storage media for execution by at least one of the one or more computer processors via at least one of the one or more computer readable memories, to perform the following operations: using at least one industrial standard to develop an ontology and a taxonomy as defining a domain of construction documents to be used for processing a plurality of engineering documents directed to a specific construction project, the ontology comprising a set of concepts and categories in the domain that shows their properties and interrelationships and the taxonomy comprising a common vocabulary to be used for terms found in the plurality of engineering documents directed to the specific construction project; receiving an input area that describes where in sample engineering documents to search for location information; training a learning model to extract location information from the input area and context information including semantic information from the sample engineering documents, using the developed ontology and taxonomy such that the extracted context information is semantically related to said taxonomy; applying the trained learning model to the plurality of engineering documents directed to the specific construction project, comprising: receiving as input data the plurality of engineering documents that comprises a data-dump of one or more three-dimensional (3D) models of the specific construction project and corresponding architecture and an overall layout for the specific construction project; and providing the input data to the trained learning model that extracts location information by automatically inferring grid coordinates in the input data even when grid coordinates in an engineering document are obscured by other content in the engineering document, wherein inferring grid coordinates includes interpolating visible information from coordinates mined from drawings of other floors in the plurality of engineering documents; upon completion of applying the trained learning model, storing each engineering document in a database of construction documents directed to the specific construction project, each engineering document being stored as annotated with its extracted location information and context information; and providing an interface that provides access to the database of construction documents directed to the specific construction project.