# US 12,008,780 B2 · System and method for data acquisition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,780 B2 - Document ID: US12008780B2 - Status: Granted - Kind: B2 - Application number: 17/815,192 - Filed: 2022-07-26 - Published: 2023-01-12 - Granted: 2024-06-11 - Earliest claimed: 2018-10-08 - Assignees: Ulc Technologies, LLC - Applicants: ULC Technologies, LLC - Inventors: Ali Asmari; Xinghua Jia; Robert E. Kodadek - Web page: https://patentgrep.com/patent/US12008780B2 ## CPC - G06T 7/593: from stereo images - B25J 9/162: {Mobile manipulator, movable base with manipulator arm mounted on it} - B25J 9/1664: {characterised by motion, path, trajectory planning} - G06T 7/337: {involving reference images or patches} - G06T 7/579: from motion - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/757: {Matching configurations of points or features} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/17: taken from planes or by drones - G06V 20/182: {Network patterns, e.g. roads or rivers} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method for pipeline data acquisition may include a software program that can autonomously review new and legacy videos collected by camera-equipped robotic systems from inside the pipelines, and automatically detect and categorize different features. Three-dimensional (3-D) point cloud may also be generated using software algorithms that stitch together like features in different video frames. ## Claims ### Claim 1 A system for data acquisition, comprising: a first transport module operable to move along a desired path, the first transport module including a first positioning system configured to provide information related to a relative position of the first transport module; a first pair of video cameras carried by the first transport module such that each of the video cameras of the first pair of video cameras is located at a different position on the first transport module and configured to capture video images as the first transport module is moving along a first path; and a first control system having at least one first processor and configured to: receive information related to the captured images from the first pair of video cameras, identify a plurality of selected features from the information received from the first pair of video cameras, determine a distance from the first pair of video cameras of the first transport module to each of the selected features for each of a plurality of frames in the captured video images, process the captured video images to generate a first three-dimensional (3-D) point cloud of the selected features, and use the first 3-D point cloud to create a map of the first path. ### Claim 2 (depends on 1) The system of claim 1, wherein the first control system is further configured to process the information related to the captured images from the first pair of video cameras using at least one simultaneous localization and mapping (SLAM) algorithm to generate the first 3-D point cloud. ### Claim 3 (depends on 2) The system of claim 2, wherein the first pair of video cameras is further configured to generate low-frequency, high-resolution images and high-frequency, low-resolution images, and the first control system is further configured to use the low-frequency, high-resolution images in a stereo SLAM process to generate the first 3-D point cloud. ### Claim 4 (depends on 3) The system of claim 3, wherein the first control system is further configured to use the high-frequency, low-resolution images in a visual SLAM process to generate a second 3-D point cloud and a first six degree-of-freedom (DOF) trajectory of the first transport module. ### Claim 5 (depends on 4) The system of claim 4, further comprising an inertial measurement unit (IMU) configured to calculate a three DOF orientation of the first transport module, and wherein the first control system is further configured to receive position information of the first transport module from the first positioning system and orientation information of the first transport module from the IMU, and to generate a second six DOF trajectory for the first transport module using the position information, the orientation information, and the first six DOF trajectory for the first transport module. ### Claim 6 (depends on 5) The system of claim 5, wherein the first control system is further configured to use the first six DOF trajectory of the first transport module to increase accuracy of the second six DOF trajectory of the first transport module. ### Claim 7 (depends on 1) The system of claim 1, further comprising: a second transport module operable to move along a desired path; a second pair of video cameras carried by the second transport module such that each of the video cameras of the second pair of video cameras is located at a different position on the second transport module and configured to capture video images as the second transport module is moving along a second path; and a second control system having at least one second processor and configured to: receive information related to the captured images from the second pair of video cameras, identify a plurality of selected features from the information received from the second pair of video cameras, determine a distance from the second transport module to each of the selected features for each of a plurality of frames in the captured video images of the second pair of video cameras, process the captured video images of the second pair of cameras to generate a three-dimensional (3-D) point cloud of the selected features related to the second path, and use the 3-D point cloud related to the second path to create a map of the second path. ### Claim 8 (depends on 7) The system of claim 7, wherein one of the first control system or the second control system is further configured to correlate the map of the first path with the map of the second path to create an integrated map of the first and second paths. ### Claim 9 (depends on 7) The system of claim 7, further comprising a second positioning system operatively connected to the second control system and configured to determine a location of the first transport module as it moves along the first path. ### Claim 10 A system for data acquisition, comprising: a first transport module operable to move along a desired path, the first transport module including a positioning system configured to provide information related to a relative position of the first transport module; a first video camera carried by the first transport module and configured to capture video images as the first transport module is moving along a first path; a first control system having at least one first processor and configured to: receive information related to the captured images from the first video camera, identify a plurality of selected features from the information received from the first video camera, process the captured video images to generate a first three-dimensional (3-D) point cloud of the selected features, and use the first 3-D point cloud to create a map of the first path; a second transport module operable to move along a desired path; a second video camera carried by the second transport module and configured to capture video images as the second transport module is moving along a second path; and a second control system having at least one second processor and configured to: receive information related to the captured images from the second video camera, identify a plurality of selected features from the information received from the second video camera, process the captured video images of the second video camera to generate a three-dimensional (3-D) point cloud of the selected features related to the second path, and use the 3-D point cloud related to the second path to create a map of the second path, and wherein at least one of the first control system or the second control system is further configured to identify data from the first path having a known relationship to data from the second path. ### Claim 11 (depends on 10) The system of claim 10, wherein the first and second control systems are each configured to process the information related to the captured images from the first and second video cameras, respectively, using at least one SLAM algorithm to generate the first and second 3-D point clouds, respectively. ### Claim 12 (depends on 10) The system of claim 10, further comprising a first positioning system configured to provide information to the first control system related to a relative position of the first transport module. ### Claim 13 (depends on 10) The system of claim 10, further comprising an IMU in communication with the second control system and configured to calculate a three DOF orientation of the first transport module. ### Claim 14 (depends on 10) The system of claim 10, wherein at least one of the first control system or the second control system is further configured to use the identified data from the first path having a known relationship to data from the second path to correlate the map of the first path with the map of the second path to create an integrated map of the first and second paths. ### Claim 15 (depends on 10) The system of claim 10, further comprising a second positioning device operatively connected to the second control system and configured to determine a location of the first transport module as it moves along the first path. ### Claim 16 (depends on 15) The system of claim 15, wherein the first transport module is part of a robotic system operable to traverse an inside of an underground pipeline, and the second transport module is operable to traverse an area above ground over the pipeline. ### Claim 17 (depends on 16) The system of claim 16, wherein the second transport module is one of an aerial vehicle, a ground vehicle, or a person. ### Claim 18 A method for data acquisition, comprising: capturing video images with a first video camera as the first video camera is moved along a first path; gathering data related to a position of the first video camera as it moves along a first path; analyzing information related to the captured images from the first video camera using at least one first processor; identifying a plurality of selected features from the information received from the first video camera using the at least one first processor; generating a first three-dimensional (3-D) point cloud of the selected features using the at least one first processor; using the first 3-D point cloud to create a map of the first path; capturing video images with a second video camera as the second video camera is moving along a second path; analyzing information related to the captured images from the second video camera using at least one second processor; identifying a plurality of selected features from the information received from the second video camera using the at least one second processor; generating a second three-dimensional (3-D) point cloud of the selected features related to the second path using the at least one second processor; using the 3-D point cloud related to the second path to create a map of the second path; and identifying data from the first path having a known relationship to data from the second path. ### Claim 19 (depends on 18) The method of claim 18, further comprising using the identified data from the first path having a known relationship to data from the second path to correlate the map of the first path with the map of the second path to create an integrated map of the first and second paths. ### Claim 20 (depends on 19) The method of claim 19, wherein at least some of the first path is through an underground structure and at least some of the second path is coincident with the first path.