# US 12,217,408 B2 · Semantic deep learning and rule optimization for surface corrosion detection and evaluation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,408 B2 - Document ID: US12217408B2 - Status: Granted - Kind: B2 - Application number: 17/572,806 - Filed: 2022-01-11 - Published: 2023-07-13 - Granted: 2025-02-04 - Earliest claimed: 2022-01-11 - Assignees: Bentley Systems, Incorporated - Applicants: Bentley Systems, Incorporated - Inventors: Zheng Yi Wu; Atiqur Rahman; Rony Kalfarisi - Web page: https://patentgrep.com/patent/US12217408B2 ## CPC - G06T 7/0004: {Industrial image inspection} - G06T 7/11: Region-based segmentation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/30136: Metal - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In various example embodiments, techniques are provided for training and/or using a semantic deep learning model, such as a segmentation-enabled CNN model, to detect corrosion and enable its quantitative evaluation. An application may include a training dataset generation tool capable of semi-automatic generation of a training dataset that includes images with labeled corrosion segments. The application may use the labeled training dataset to train a semantic deep learning model to detect and segment corrosion in images of an input dataset at the pixel-level. The application may apply an input dataset to the trained semantic deep learning model to produce a semantically segmented output dataset that includes labeled corrosion segments. The application may include an evaluation tool that quantitatively evaluates corrosion in the semantically segmented output dataset, to allow severity of the corrosion to be classified. ## Claims ### Claim 1 A method for training and/or using a semantic deep learning model to detect surface corrosion, comprising: receiving, by a training dataset generation tool of an application executing on one or more computing devices, a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion; applying, by the training dataset generation tool, an unsupervised image segmentation algorithm to segment a first portion of the images of the training dataset; prompting a user to manually label at least some of the segments of the first portion as corrosion segments to produce labeled images; optimizing, by the training dataset generation tool, classification rules based on the labeled images of the first portion to produce a rule-based classifier; applying, by the training dataset generation tool, the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion as corrosion segments; and using the labeled training dataset to train the semantic deep learning model to detect and segment corrosion in images of an input dataset. ### Claim 2 (depends on 1) The method of claim 1, wherein the first portion includes a smaller number of images than the second portion. ### Claim 3 (depends on 1) The method of claim 1, wherein the unsupervised image segmentation algorithm is a factor-based texture segmentation (FSEG) algorithm. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of images are red-green-blue (RGB) images, the classification rules are RGB color channel-based classification rules and the rule-based classifier is a RGB color channel-based classifier. ### Claim 5 (depends on 1) The method of claim 1, wherein the semantic deep learning model is a semantic segmentation-enabled convolutional neural network (CNN) model. ### Claim 6 (depends on 1) The method of claim 1, wherein the using the labeled training dataset to train the semantic deep learning model comprises: minimizing a total loss function of the semantic deep learning model that weights together misclassification of corrosion pixels and misclassification of non-corrosion pixels. ### Claim 7 (depends on 1) The method of claim 1, further comprising: applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments. ### Claim 8 (depends on 7) The method of claim 7, further comprising: displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, indications of the corrosion segments. ### Claim 9 (depends on 7) The method of claim 7, further comprising: calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment's area. ### Claim 10 (depends on 9) The method of claim 9, further comprising: dividing, by the application, corrosion segments into categories based on a comparison of the degree of corrosion index of each corrosion segment to one or more thresholds. ### Claim 11 (depends on 10) The method of claim 10, further comprising: displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, at least one of the degree of corrosion index or the category of each corrosion segment. ### Claim 12 A method for training and/or using a semantic deep learning model to detect surface corrosion, comprising: training, by an application executing on one or more computing devices, the semantic deep learning model to detect and segment corrosion in images of an input dataset; applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments; calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment's area; and displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices, the degree of corrosion index of each corrosion segment. ### Claim 13 (depends on 12) The method of claim 12, further comprising: dividing, by the application, corrosion segments into categories based on a comparison of the degree of corrosion index of each corrosion segment to one or more thresholds. ### Claim 14 (depends on 13) The method of claim 13, further comprising: displaying, by the application, in a user interface or storing to memory/storage of the one or more computing devices the category of each corrosion segment. ### Claim 15 (depends on 12) The method of claim 12, wherein the training further comprises: receiving, by a training dataset generation tool of the application, a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion; applying, by the training dataset generation tool, an unsupervised image segmentation algorithm to segment a first portion of the images of the training dataset; prompting a user to manually label at least some of the segments of the first portion as corrosion segments to produce labeled images; optimizing, by the training dataset generation tool, classification rules based on the labeled images of the first portion to produce a rule-based classifier; applying, by the training dataset generation tool, the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion as corrosion segments; and using the labeled training dataset to train the semantic deep learning model. ### Claim 16 (depends on 15) The method of claim 15, wherein the first portion includes a smaller number of images than the second portion. ### Claim 17 (depends on 15) The method of claim 15, wherein the unsupervised image segmentation algorithm is a factor-based texture segmentation (FSEG) algorithm, the plurality of images are red-green-blue (RGB) images, the classification rules are RGB color channel-based classification rules, the rule-based classifier is a RGB color channel-based classifier, and the semantic deep learning model is a semantic segmentation-enabled convolutional neural network (CNN) model. ### Claim 18 A non-transitory electronic device readable medium having instructions stored thereon that when executed on one or more processors of one or more electronic devices are operable to: receive a training dataset that includes a plurality of images of infrastructure having at least some surface corrosion; apply an image segmentation algorithm to segment a first portion of the images of the training dataset; receive user-provided labels for at least some of the segments of the first portion to produce labeled images; optimize classification rules based on the labeled images of the first portion to produce a rule-based classifier; apply the rule-based classifier to a second portion of the images of the training dataset to automatically segment and label at least some of the segments of the second portion; and use the labeled training dataset to train a semantic deep learning model to detect and segment corrosion in images of an input dataset. ### Claim 19 (depends on 18) The non-transitory electronic device readable medium of claim 18, further comprising: applying an input dataset that includes a plurality of images of infrastructure to the trained semantic deep learning model to produce a semantically segmented output dataset, the semantically segmented output dataset including labeled corrosion segments. ### Claim 20 (depends on 19) The non-transitory electronic device readable medium of claim 19, further comprising: calculating a degree of corrosion index for each corrosion segment based on a mean grayscale value of pixels within the corrosion segment.