# US 11,310,417 B2 · State estimation and localization for ROV-based structural inspection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,417 B2 - Document ID: US11310417B2 - Status: Granted - Kind: B2 - Application number: 16/610,647 - Filed: 2018-05-03 - Published: 2020-02-27 - Granted: 2022-04-19 - Earliest claimed: 2017-05-04 - Assignees: Carnegie Mellon University - Applicants: Carnegie Mellon University - Inventors: Nathan Michael; Tabitha Edith Lee; Curtis Boirum - Web page: https://patentgrep.com/patent/US11310417B2 ## CPC - G21C 17/01: Inspection of the inner surfaces of vessels - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - G01S 5/16: using electromagnetic waves other than radio waves - G01S 5/163: {Determination of attitude (using inertial means G01C9/00; control of attitude G05D1/49)} - G06T 7/207: for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53) - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/74: {involving reference images or patches} - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G21C 17/013: Inspection vehicles - G21D 3/001: {Computer implemented control} - H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects - Y02E 30/00: Energy generation of nuclear origin - Y02E 30/30: Nuclear fission reactors - G21C: NUCLEAR REACTORS (fusion reactors, hybrid fission-fusion reactors G21B; nuclear explosives G21J) - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G21D: NUCLEAR POWER PLANT - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION ## Abstract A vision-based state estimation framework to estimate the state of an underwater remotely operated vehicle (ROV) used for inspection of an underwater structure, for example, a nuclear reactor pressure vessel. The framework employs an external overhead, pan-tilt-zoom (PTZ) camera as the primary sensing modality and incorporates prior knowledge of the geometry of the structure. ## Claims ### Claim 1 A system for localizing an underwater remotely controlled vehicle moving within a structure filled at least partially with water, comprising: a map of the structure, the map indicating locations of one or more landmarks defined on the structure; a pan-tilt-zoom camera, disposed in the water at a stationary position with respect to the structure; software, executing on a processor, the software iterating a loop performing the functions of: panning, tilting and zooming the camera to follow a plurality of fiducials mounted on the vehicle; collecting an image from the camera; estimating an angular velocity of the camera about a vector comprising a combination of the pan and tilt axes of the camera; predicting a future state of the system based on a current state of the system and the estimate of the angular velocity; calculating a correction to the predicted state of the system based on detecting in the collected image, the plurality of fiducials on the vehicle and one or more of the landmarks indicated on the map of the structure; applying the correction to the predicted state of the system; and using the corrected predicted state as a previous state for a next iteration of the loop; wherein the predicted state of the system includes at least a position and an orientation of the vehicle with respect to the structure and current pan, tilt and zoom settings of the camera. ### Claim 2 (depends on 1) The system of claim 1 wherein predicting the future state of the system is further based on one or more additional terms representing the focal length of the camera. ### Claim 3 (depends on 1) The system of claim 1 wherein predicting the future state of the system produces a predicted state and a covariance of the predicted state. ### Claim 4 (depends on 3) The system of claim 3 wherein applying the corrected state to the predicted state produces a corrected state and a covariance of the corrected state. ### Claim 5 (depends on 1) The system of claim 1 wherein the estimate of the angular velocity of the camera is based on a comparison of the collected image with one or more previous images. ### Claim 6 (depends on 5) The system of claim 5 wherein estimating the angular velocity of the camera uses image registration in a homography-based method. ### Claim 7 (depends on 6) The system of claim 6 wherein the image registration is intensity-based. ### Claim 8 (depends on 1) The system of claim 1 wherein the estimate of the angular velocity of the camera is based on pan and tilt angles reported by the camera. ### Claim 9 (depends on 1) The system of claim 1 wherein detecting the one or more landmarks defined on the structure from the collected image uses the map of the structure and a previous state of the system as inputs. ### Claim 10 (depends on 1) The system of claim 1 wherein the software performs the further function of detecting the one or more landmarks defined on the structure and updating the map of the structure by updating a position of the one or more detected landmarks. ### Claim 11 (depends on 10) The system of claim 10 wherein the one or more landmarks are defined points or lines on the structure. ### Claim 12 (depends on 11) The system of claim 11 wherein the map of the structure consists of a series of intersecting planes. ### Claim 13 (depends on 12) The system of claim 12 wherein the lines represent the intersections between the planes. ### Claim 14 (depends on 1) The system of claim 1 wherein the software performs the further function of determining an initial position and initial orientation of the camera. ### Claim 15 (depends on 1) The system of claim 1 wherein the software performs the further function of determining an initial location of the vehicle. ### Claim 16 (depends on 1) The system of claim 1 wherein the software performs the further function of suspending tracking and attempting to re-acquire the vehicle when tracking of the vehicle is determined to have been lost. ### Claim 17 (depends on 4) The system of claim 4 wherein the software uses an extended Kalman filter to determine the predicted state and the covariance of the predicted state and the corrected state and the covariance of the corrected state. ### Claim 18 (depends on 1) The system of claim 1 wherein the camera is enclosed in a watertight housing. ### Claim 19 (depends on 18) The system of claim 18 wherein the housing is disposed at a fixed location relative to the structure and the vehicle. ### Claim 20 (depends on 1) The system of claim 1 wherein the detected landmarks are projected from a frame representing the structure to a frame representing an optical center of the camera. ### Claim 21 (depends on 1) The system of claim 1 wherein the one or more landmarks are detected in the image using feature detection based on geometric shape.