# US 10,480,939 B2 · High speed stereoscopic pavement surface scanning system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,480,939 B2 - Document ID: US10480939B2 - Status: Granted - Kind: B2 - Application number: 16/257,384 - Filed: 2019-01-25 - Published: 2019-05-23 - Granted: 2019-11-19 - Earliest claimed: 2016-01-15 - Assignees: Fugro Roadware Inc. - Applicants: Fugro Roadware Inc. - Inventors: Andrea Annovi; Ryan Fechney; Shashibushan Yenkanchi; David Lowe; Hitesh Shah; Prasanna Kumar Sivakumar; Ishar Pratap Singh; Miroslava Galchinsky - Web page: https://patentgrep.com/patent/US10480939B2 ## CPC - G01C 7/04: involving a vehicle which moves along the profile to be traced - E01C 23/01: Devices or auxiliary means for setting-out or checking the configuration of new surfacing, e.g. templates, screed {or reference line} supports (form rails E01C19/50); Applications of apparatus for measuring, indicating, or recording the surface configuration of existing surfacing, e.g. profilographs (E01C23/07 takes precedence; measuring roughness or irregularity in general G01B) - E01C 23/07: Apparatus combining measurement of the surface configuration of paving with application of material in proportion to the measured irregularities (measuring means only E01C23/01 {G01B}) - E01C 23/08: for roughening or patterning; for removing {the surface down to a predetermined depth} high spots or material bonded to the surface, e.g. markings; {for maintaining earth roads, clay courts or like surfaces by means of surface working tools, e.g. scarifiers, levelling blades}(removing matter not bonded to the surface E01H1/00; roughening or detaching ice E01H5/12) - G01B 11/24: for measuring contours or curvatures - G01B 11/303: {using photoelectric detection means} - G06T 7/0004: {Industrial image inspection} - G06T 7/593: from stereo images - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/296: Synchronisation thereof; Control thereof - H04N 23/56: provided with illuminating means - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30132: Masonry; Concrete - G06T 2207/30248: Vehicle exterior or interior - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)} - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - E01C: CONSTRUCTION OF, OR SURFACES FOR, ROADS, SPORTS GROUNDS, OR THE LIKE; MACHINES OR AUXILIARY TOOLS FOR CONSTRUCTION OR REPAIR (forming roads or like surfaces by compacting or grading snow or ice E01H) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Disclosed is a mobile pavement surface scanning system for detecting pavement distress. In an embodiment the system comprises one or more light sources mounted on the mobile vehicle for illuminating a pavement, one or more stereoscopic image capturing devices mounted on the vehicle for capturing sequential images of an illuminated pavement surface, and a plurality of positioning sensors mounted on the mobile vehicle, the positioning sensors adapted to encode movement of the mobile vehicle and provide a synchronization signal for the sequential images captured by the one or more stereoscopic image capture devices. One or more computer processors are adapted to synchronize the intensity image pairs captured by each camera in the one or more stereoscopic image capturing devices, perform a 3D reconstruction of the pavement from the intensity image pairs using stereoscopic principles, generate a depth image and an intensity image pair from the 3D reconstruction, and process at least one of the depth image and the intensity image utilizing one or more distress detection modules to detect a type of pavement distress. ## Claims ### Claim 1 A mobile pavement surface scanning system for detecting pavement distress, comprising: one or more light sources mounted to a mobile vehicle for illuminating a pavement surface; one or more stereoscopic image capturing devices mounted to the mobile vehicle for capturing sequential images of the illuminated pavement surface; a plurality of positioning sensors mounted to the mobile vehicle, the positioning sensors adapted to encode movement of the mobile vehicle and provide a synchronization signal for the sequential images captured by the one or more stereoscopic image capture devices; and one or more computer processors configured to: synchronize the sequential images captured by each camera of the one or more stereoscopic image capturing devices; generate intensity image pairs from the synchronized sequential images; perform a 3D reconstruction of the illuminated pavement surface from the intensity image pairs using stereoscopic principles; generate a depth image and an intensity image pair from the 3D reconstruction; and process at least one of the depth image and the intensity image utilizing one or more distress detection modules to detect a type of pavement distress. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more distress detection modules comprise a computer vision module for detecting pavement distress utilizing at least one of the depth image and an intensity image pair. ### Claim 3 (depends on 2) The system of claim 2, wherein the one or more distress detection modules further comprise a machine learning module, and the computer vision module is adapted to generate a learning feed forward to the machine learning module. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more distress detection modules comprise a machine learning module for detecting pavement distress utilizing at least one of the depth image and an intensity image pair. ### Claim 5 (depends on 4) The system of claim 4, wherein the machine learning module includes a learning feedback loop to enable the machine learning module to improve detection of pavement distresses. ### Claim 6 (depends on 5) The system of claim 5, wherein the machine learning module comprises an artificial intelligence (AI) engine executing a learning algorithm to detect and classify distresses based on its iterative training. ### Claim 7 (depends on 6) The system of claim 6, wherein the machine learning module is adapted to provide a feedback signal to dynamically change a parameter of a component on the mobile vehicle for capturing the sequential images on the illuminated pavement surface. ### Claim 8 (depends on 7) The system of claim 7, wherein the feedback signal is processed to change a parameter of the one or more light sources mounted to the mobile vehicle. ### Claim 9 (depends on 7) The system of claim 7, wherein the feedback signal is processed to change a parameter of the one or more stereoscopic image capturing devices. ### Claim 10 (depends on 6) The system of claim 6, wherein the machine learning module is adapted to select a type of image processing filter in dependence upon the type of pavement distress being detected. ### Claim 11 (depends on 10) The system of claim 10, wherein the machine learning module is adapted to categorize the type of pavement distress detected, and to store the geo-reference for the detected pavement distress in real time as the survey data is stored. ### Claim 12 A method of scanning a pavement surface for detecting pavement distress, comprising: providing one or more light sources mounted to a mobile vehicle for illuminating a pavement surface; providing one or more stereoscopic image capturing devices mounted to the mobile vehicle for capturing sequential images of the illuminated pavement surface; providing a plurality of positioning sensors mounted to the mobile vehicle, the positioning sensors adapted to encode movement of the mobile vehicle and provide a synchronization signal for the sequential images captured by the one or more stereoscopic image capture devices; and providing one or more computer processors configured to: synchronize the sequential images captured by each camera of the one or more stereoscopic image capturing devices; generate intensity image pairs from the synchronized sequential images; perform a 3D reconstruction of the illuminated pavement surface from the intensity image pairs using stereoscopic principles; generate a depth image and an intensity image pair from the 3D reconstruction; and process at least one of the depth image and the intensity image utilizing one or more distress detection modules to detect a type of pavement distress. ### Claim 13 (depends on 12) The method of claim 12, wherein the one or more distress detection modules comprise a computer vision module for detecting pavement distress utilizing at least one of the depth image and an intensity image pair. ### Claim 14 (depends on 13) The method of claim 13, wherein the one or more distress detection modules further comprise a machine learning module, and the computer vision module is adapted to generate a learning feed forward to the machine learning module. ### Claim 15 (depends on 12) The method of claim 12, wherein the one or more distress detection modules comprise a machine learning module for detecting pavement distress utilizing at least one of the depth image and an intensity image pair. ### Claim 16 (depends on 15) The method of claim 15, wherein the machine learning module includes a learning feedback loop to enable the machine learning module to improve detection of pavement distresses. ### Claim 17 (depends on 16) The method of claim 16, wherein the machine learning module comprises an artificial intelligence (AI) engine executing a learning algorithm to detect and classify distresses based on its iterative training. ### Claim 18 (depends on 17) The method of claim 17, wherein the machine learning module is adapted to provide a feedback signal to dynamically change a parameter of a component on the mobile vehicle for capturing the sequential images on the illuminated pavement surface. ### Claim 19 (depends on 18) The method of claim 18, wherein the feedback signal is processed to change a parameter of the one or more light sources mounted to the mobile vehicle. ### Claim 20 (depends on 18) The method of claim 18, wherein the feedback signal is processed to change a parameter of the one or more stereoscopic image capturing devices. ### Claim 21 (depends on 17) The method of claim 17, wherein the machine learning module is adapted to select a type of image processing filter in dependence upon the type of pavement distress being detected. ### Claim 22 (depends on 21) The method of claim 21, wherein the machine learning module is adapted to categorize the type of pavement distress detected, and to store the geo-reference for the detected pavement distress in real time as the survey data is stored.