# US 2022/0044146 A1 · Techniques for extracting machine-readable information from p&ids

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044146 A1
- Document ID: US20220044146A1
- Status: Published application
- Kind: A1
- Application number: 17/129,205
- Filed: 2020-12-21
- Published: 2022-02-10
- Earliest claimed: 2020-08-04
- Assignees: Bentley Systems, Incorporated
- Applicants: Bentley Systems, Incorporated
- Inventors: Marc-André Gardner; Karl-Alexandre Jahjah
- Web page: https://patentgrep.com/patent/US20220044146A1

## CPC

- G06N 20/00: Machine learning
- G06V 30/422: Technical drawings; Geographical maps
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F 40/242: Dictionaries
- G06N 5/04: Inference or reasoning models
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 30/10: Character recognition
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In example embodiments, techniques are provided for using machine learning to extract machine-readable labels for text boxes and symbols in P&IDs in image-only formats. A P&ID data extraction application uses an optical character recognition (OCR) algorithm to predict labels for text boxes in a P&ID. The P&ID data extraction application uses a first machine learning algorithm to detect symbols in the P&ID and return a predicted bounding box and predicted class of equipment for each symbol. One or more of the predicted bounding boxes may be decimate by non-maximum suppression to avoid overlapping detections. The P&ID data extraction application uses a second machine learning algorithm to infer properties for each detected symbol having a remaining predicted bounding box. The P&ID data extraction application stores the predicted bounding box and a label including the predicted class of equipment and inferred properties in a machine-readable format.

## Claims

### Claim 1

A method for extracting machine-readable labels for text boxes and symbols in piping and instrumentation drawings (P&IDs) in image-only formats, comprising:
predicting, by an optical character recognition (OCR) algorithm of a P&ID data extraction application executing on a computing device, bounding boxes and text for text boxes in a P&ID in an image-only format;
detecting, by a first machine learning algorithm of the P&ID data extraction application, symbols in the P&ID, the detecting to return a predicted bounding box and predicted class of equipment for each symbol;
decimating one or more of the predicted bounding boxes to avoid overlapping detections;
inferring, by a second machine learning algorithm of the P&ID data extraction application, properties for each detected symbol having a remaining predicted bounding box; and
storing, in a machine-readable format in a memory of the computing device, for each detected symbol having a remaining predicted bounding box, the predicted bounding box and a label including the predicted class of equipment and inferred properties.

### Claim 2 (depends on 1)

The method of claim 1, wherein the detecting produces a confidence in each predicted bounding box and associated predicted label, and the inferring produces a confidence in each property, and the storing stores each confidence.

### Claim 3 (depends on 1)

The method of claim 1, further comprising: preprocess, by the P&ID data extraction application, the P&ID to at least one of rasterize, resize or binarize the P&ID.

### Claim 4 (depends on 1)

The method of claim 1, wherein the OCR algorithm uses a domain-specific dictionary in language modeling, the domain-specific dictionary based on knowledge of words and codes used in P&IDs in a given industry or based on a particular list of words and codes known to be used by a specific user in P&IDs.

### Claim 5 (depends on 1)

The method of claim 1, wherein the OCR algorithm uses domain-specific patterns of n-grams in language modeling to prevent bias against words and codes expected in P&IDs.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first machine learning algorithm of the P&ID data extraction is based on a region-based convolutional neural network object detection architecture.

### Claim 7 (depends on 6)

The method of claim 6, wherein the region-based convolutional neural network object detection architecture is a Faster Region-based Convolutional Neural Network (R-CNN) architecture.

### Claim 8 (depends on 6)

The method of claim 6, wherein the region-based convolutional neural network object detection architecture is adapted to use enlarged image dimensions, use an increased number of filters in an input layer and a decreased number of filters in an output layer, consider an expanded number of resolutions, to give higher weights to samples that are part of uncommon classes in a loss calculation or to use an increased number of proposal bounding boxes and predicted bounding boxes.

### Claim 9 (depends on 1)

The method of claim 1, wherein the decimating uses a non-maximum suppression algorithm.

### Claim 10 (depends on 9)

The method of claim 9, wherein the non-maximum suppression algorithm includes operations for:
weighting confidences of each bounding box based on a measure of importance of an associated class and location;
computing a dynamic overlap of other bounding boxes and a bounding box with highest weighted confidence; and
deleting bounding boxes with a calculated dynamic overlap greater than a predetermined threshold.

### Claim 11 (depends on 1)

The method of claim 1, wherein the second machine learning algorithm of the P&ID data extraction is based on a deep neural network architecture adapted for properties classification.

### Claim 12 (depends on 1)

The method of claim 1, wherein inferring produces a variable number of properties for each symbol, and each property has a variable number of property values.

### Claim 13 (depends on 1)

The method of claim 1, wherein the machine-readable format is a JavaScript Object Notation (JSON) file associated with the P&ID.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
displaying the predicted labels for the text boxes and symbols in a user interface of the P&ID data extraction application.

### Claim 15

A computing device comprising:
a display screen;
a processor; and
a memory coupled to the processor and configured to store a piping and instrumentation drawing (P&ID) in image-only format and a P&ID data extraction application, the P&ID data extraction application when executed operable to:
predict using optical character recognition (OCR) bounding boxes and text for text boxes in the P&ID,
detect using machine learning symbols in the P&ID and return a predicted io bounding box and predicted class of equipment for each detected symbol,
infer using machine learning properties for each detected symbol, and
store, for each detected symbol, the predicted bounding box and a label including the predicted class of equipment and inferred properties in a machine-readable format in the memory.

### Claim 16 (depends on 15)

The computing device of claim 15, wherein the P&ID data extraction application when executed is further operable to produce a confidence in each predicted bounding box and associated predicted label, produce a confidence in each property, and store each confidence in the memory.

### Claim 17

A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:
predict, using optical character recognition (OCR), bounding boxes and text for text boxes in a piping and instrumentation drawing (P&ID) in an image-only format;
detect, using machine learning, symbols in the P&ID and return a predicted bounding box and predicted class of equipment for each symbol;
decimate one or more of the predicted bounding boxes to avoid overlapping detections;
infer, using machine learning, properties for each detected symbol having a remaining predicted bounding box; and
store in a machine-readable format, for each detected symbol having a remaining predicted bounding box, the predicted bounding box and a label including the predicted class of equipment and inferred properties.

### Claim 18 (depends on 17)

The non-transitory electronic-device readable medium of claim 17, wherein the instructions when executed are further operable to:
produce a confidence in each predicted bounding box and associated predicted label, produce a confidence in each property, and store each confidence.

### Claim 19 (depends on 17)

The non-transitory electronic-device readable medium of claim 17, wherein the instructions when executed are further operable to:
preprocess the P&ID to at least one of rasterize, resize or binarize the P&ID.

### Claim 20 (depends on 16)

The non-transitory electronic-device readable medium of claim 16, wherein the OCR uses at least one of a domain-specific dictionary or domain-specific patterns of n-grams in language modeling, the machine learning used in detecting symbols employs a region-based convolutional neural network object detection architecture, the decimation uses a non-maximum suppression algorithm, and the machine learning used in inferring properties uses a deep neural network architecture.
