# US 12,222,298 B2 · Multi-sensor pipe inspection system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,298 B2 - Document ID: US12222298B2 - Status: Granted - Kind: B2 - Application number: 17/992,567 - Filed: 2022-11-22 - Published: 2023-03-16 - Granted: 2025-02-11 - Earliest claimed: 2018-05-04 - Assignees: Hydromax USA, LLC - Applicants: Hydromax USA, LLC - Inventors: Guy Schiavone; Adam Fredrick Stewart - Web page: https://patentgrep.com/patent/US12222298B2 ## CPC - G01B 21/20: for measuring contours or curvatures, e.g. determining profile - G01N 21/954: Inspecting the inner surface of hollow bodies, e.g. bores - E21B 47/002: by visual inspection - G01M 3/005: {using pigs or moles (G01M3/246, G01M3/2823 take precedence)} - G01M 3/007: {Leak detector calibration, standard leaks (G01M3/207 takes precedence)} - G01M 3/38: by using light (G01M3/02 takes precedence) - G01N 21/8507: {Probe photometers, i.e. with optical measuring part dipped into fluid sample} - G01N 21/8803: {Visual inspection (measuring projectors G01B9/08)} - G01N 21/8851: {Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges (G01N21/8806 and G01N21/93 - G01N21/95692 take precedence; optical measurement of dimensions G01B11/00; optical scanning G02B26/10; image transformation G06T3/00; computerised image enhancement G06T5/00; image processing per se for flaw detection G06T7/0002)} - G06T 11/00: Two-dimensional [2D] image generation - H04N 7/181: {for receiving images from a plurality of remote sources} - E03F 7/00: Other installations or implements for operating sewer systems, e.g. for preventing or indicating stoppage; Emptying cesspools - F16L 55/00: Devices or appurtenances for use in, or in connection with, pipes or pipe systems (F16L1/00 - F16L53/00, F16L57/00, F16L59/00 take precedence; repairing or joining pipes on or under water F16L1/26; cleaning of pipes B08B9/02, e.g. removal of blockages B08B9/027; devices for preventing bursting of water pipes by freezing E03B7/10) - F16L 2201/30: Detecting leaks - G01N 2021/8874: {Taking dimensions of defect into account} - G01N 2021/888: {Marking defects} - G01N 2021/8893: {providing a video image and a processed signal for helping visual decision} - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - E21B: EARTH OR ROCK DRILLING; OBTAINING OIL, GAS, WATER, SOLUBLE OR MELTABLE MATERIALS OR A SLURRY OF MINERALS FROM WELLS - G01M: TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - E03F: SEWERS; CESSPOOLS - F16L: PIPES; JOINTS OR FITTINGS FOR PIPES; SUPPORTS FOR PIPES, CABLES OR PROTECTIVE TUBING; MEANS FOR THERMAL INSULATION IN GENERAL ## Abstract An approach for collecting disparate data within a pipe involves a sensor arrangement configured to be deployed within the pipe. The sensor arrangement includes a plurality of sensors configured to detect disparate data related to the pipe. Each sensor of the plurality of sensors is coupled to a respective collection computer on the sensor arrangement. A synchronization module is configured to synchronize the disparate data. A database is configured to store the synchronized data. A processor is configured to process the synchronized data. A user interface configured to present the synchronized data to a user. ## Claims ### Claim 1 A system for collecting disparate data within a pipe, comprising: at least one sensor configured to be deployed within the pipe, wherein the at least one sensor generates data associated with at least one of structural and material defects of the pipe; a processor that executes instructions to process the data generated from the at least one sensor, wherein the instructions include: process preliminary data from the at least one sensor, wherein the instructions to process the preliminary data from the at least one sensor includes additional instructions to: apply a water-level mask by blanking above a water line; apply a bilateral filter to reduce noise in the data; implement a binarization threshold to the data filtered by the bilateral filter; extract contour features from the data generated from the at least one sensor; match some of the data to a template of the pipe; determine deformations in the pipe based on the data generated from the at least one sensor; and analyze quality of the data generated from the at least one sensor; wherein a corrective action is enabled to be performed at a defective location of the at least one of structural and material defects of the pipe, and wherein the defective location is determined based on the data processed by the processor. ### Claim 2 (depends on 1) The system of claim 1, wherein the instructions apply a bilateral filter to reduce noise in the data further includes the additional instructions to: establish a space domain of all possible positions in an image kernel; generate a spatial Gaussian kernel that is a weighted average of the intensity of pixels centered around a center pixel; and apply a multiplicative factor to preserves edges and blur noise. ### Claim 3 (depends on 1) The system of claim 1, wherein the instructions to extract contour features from the data generated from the at least one sensor further includes the additional instructions to: segment an image into regions based on perceptual similarity. ### Claim 4 (depends on 3) The system of claim 3, wherein the segmentation of the image into regions based on perceptual similarity is accomplished by instructions to: determine connected closed contours or chains of pixels that surround regions of similar pixels in the image, wherein closed contours are numbered sequentially, and create a hierarchy for holes in the image holes and for regions within the holes, recursively. ### Claim 5 (depends on 4) The system of claim 4, wherein the segmentation of the image into regions based on perceptual similarity is accomplished by instructions to: scan rows in the image; and scan regions in the image having like pixels in counter-clockwise manner. ### Claim 6 (depends on 3) The system of claim 3, wherein the segmentation of the image into regions based on perceptual similarity is accomplished by instructions to: define a bounding box around the largest span of connected closed contours or chains of pixels. ### Claim 7 (depends on 1) The system of claim 1, wherein the instructions to determine deformations in the pipe based on the data generated from the at least one sensor further includes the additional instructions to: determine a minimum diameter or interior dimension of the pipe. ### Claim 8 (depends on 1) The system of claim 1, wherein the instructions to analyze quality of the data generated from the at least one sensor further includes the additional instructions to: provide a view containing both GIS data and preprocessed data; and determine whether the GIS data and preprocessed data are linked, if proper linkage is determined, then inspection data is cleared for further manual inspection and if proper linkage is not found, then the data is marked for corrective action. ### Claim 9 A system for collecting disparate data within a pipe, comprising: at least one sensor configured to be deployed within the pipe, wherein the at least one sensor generates data associated with at least one of structural and material defects of the pipe; a processor that executes instructions to process the data generated from the at least one sensor, wherein the instructions include: process the data from the at least one sensor; extract contour features from the data generated from the at least one sensor, wherein the instructions to extract contour features from the data generated from the at least one sensor includes additional instructions to segment an image into regions based on perceptual similarity, wherein the segmentation of the image into regions based on perceptual similarity is accomplished by instructions to define a bounding box around the largest span of connected closed contours or chains of pixels; compute angles of sediment contour segments; determine a presence of unconnected sediment segments; and extend any unconnected sediment segments to form an unbroken sediment surface; match some of the data to a template of the pipe; detect sediment depth or volume within the pipe based on the data generated from the at least one sensor; determine deformations in the pipe based on the data generated from the at least one sensor; and analyze quality of the data generated from the at least one sensor; wherein a corrective action is enabled to be performed at a defective location of the at least one of structural and material defects of the pipe, and wherein the defective location is determined based on the data processed by the processor. ### Claim 10 A system for collecting disparate data within a pipe, comprising: at least one sensor configured to be deployed within the pipe, wherein the at least one sensor generates data associated with at least one of structural and material defects of the pipe; a processor that executes instructions to process the data generated from the at least one sensor, wherein the instructions include: process the data from the at least one sensor; extract contour features from the data generated from the at least one sensor; match some of the data to a template of the pipe, wherein the instructions to match some of the data to the template of the pipe includes additional instructions to: determine a center of the pipe by centering an image in a pixel canvas, wherein for an N-pixel×N-pixel image, the pixel canvas used is 3N-pixel×3N-pixel, and wherein N is any integer; detect sediment depth or volume within the pipe based on the data generated from the at least one sensor; determine deformations in the pipe based on the data generated from the at least one sensor; and analyze quality of the data generated from the at least one sensor; wherein a corrective action is enabled to be performed at a defective location of the at least one of structural and material defects of the pipe, and wherein the defective location is determined based on the data processed by the processor. ### Claim 11 A system for collecting disparate data within a pipe, comprising: at least one sensor configured to be deployed within the pipe, wherein the at least one sensor generates data associated with at least one of structural and material defects of the pipe; a processor that executes instructions to process the data generated from the at least one sensor, wherein the instructions include: process the data from the at least one sensor; extract contour features from the data generated from the at least one sensor; match some of the data to a template of the pipe; detect sediment depth or volume within the pipe based on the data generated from the at least one sensor, wherein the instructions to detect sediment depth or volume within the pipe based on the data generated from the at least one sensor includes additional instructions to: construct a mask to blank out everything outside of a pipe wall perimeter, wherein a mask radius smaller than a radius of the pipe radius, classify every non-zero pixel inside of the mask as sediment, calculate a horizon line of the sediment; determine deformations in the pipe based on the data generated from the at least one sensor; and analyze quality of the data generated from the at least one sensor; wherein a corrective action is enabled to be performed at a defective location of the at least one of structural and material defects of the pipe, and wherein the defective location is determined based on the data processed by the processor. ### Claim 12 (depends on 11) The system of claim 11, wherein the instructions to detect sediment depth or volume within the pipe based on the data generated from the at least one sensor further includes the additional instructions to: select interior points of the sediment below the horizon line at various intervals; and employ at least one of (a) a regression spline and (b) a piecewise linear regression line. ### Claim 13 (depends on 12) The system of claim 12, wherein the instructions to detect sediment depth or volume within the pipe based on the data generated from the at least one sensor further includes the additional instructions to: apply a Random Sample Consensus (RANSAC) to eliminate the noise outliers. ### Claim 14 (depends on 13) The system of claim 13, wherein the instructions to detect sediment depth or volume within the pipe based on the data generated from the at least one sensor further includes the additional instructions to: create a visualization of a surface of the sediment via polgonization. ### Claim 15 (depends on 14) The system of claim 14, wherein the instructions to detect sediment depth or volume within the pipe based on the data generated from the at least one sensor further includes the additional instructions to: calculate a surface area of the surface of the sediment via a pixel flood-fill between the horizon line of the sediment and the pipe wall perimeter.