# US 10,121,091 B2 · IMU-aided image registration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,091 B2 - Document ID: US10121091B2 - Status: Granted - Kind: B2 - Application number: 15/428,638 - Filed: 2017-02-09 - Published: 2018-08-09 - Granted: 2018-11-06 - Earliest claimed: 2017-02-09 - Assignees: Rosemount Aerospace Inc. - Applicants: Rosemount Aerospace Inc. - Inventors: Todd Anthony Ell - Web page: https://patentgrep.com/patent/US10121091B2 ## CPC - G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration - G01C 19/5776: Signal processing not specific to any of the devices covered by groups G01C19/5607 - G01C19/5719 - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/73: using feature-based methods - G06T 2207/10004: Still image; Photographic image - G06T 2207/10016: Video; Image sequence - G06T 2207/30212: Military - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) ## Abstract An imager device disposed on a moving body captures first image data at a first time and second image data at a second, subsequent time. An inertial measurement unit (IMU) disposed on the moving body senses motion of the moving body between the first time and the second time. The first image data is registered to the second image data based on inertial measurement data corresponding to the sensed motion to produce first registered image data. In response to identifying that image features are common to both the first registered image data and the second image data, the first registered image data is registered to the second image data based on the identified common features to produce output registered image data. In response to determining that no image features are common to both the first registered image data and the second image data, the first registered image data is output. ## Claims ### Claim 1 A method comprising: receiving first image data that is captured at a first time by an imager device disposed on a moving body; receiving second image data that is captured by the imager device at a second time that is subsequent to the first time; receiving inertial measurement data sensed by an inertial measurement unit (IMU) disposed on the moving body, the inertial measurement data corresponding to sensed motion of the moving body between the first time and the second time; registering the first image data to the second image data based on the received inertial measurement data to produce first registered image data; in response to identifying that image features are common to both the first registered image data and the second image data: registering the first registered image data to the second image data based on the identified features that are common to both the first registered image data and the second image data to produce second registered image data; and outputting the second registered image data as output registered image data; and outputting the first registered image data as the output registered image data in response to determining that no image features are common to both the first registered image data and the second image data; identifying a target within the second image data and the output registered image data; determining a line-of-sight vector from the imager device to the target based on pixel locations of the identified target within the second image data and the output registered image data; and providing the line-of-sight vector to a guidance control system configured to control motion of the moving body; and modifying a trajectory of the moving body based on the line-of-sight vector provided to the guidance control system. ### Claim 2 (depends on 1) The method of claim 1, wherein registering the first image data to the second image data based on the received inertial measurement data comprises applying translation and rotation operations to pixel locations of the first image data based on the received inertial measurement data. ### Claim 3 (depends on 2) The method of claim 2, wherein the translation operations correspond to pitching and yawing motion of the moving body between the first time and the second time; and wherein the rotation operations correspond to rolling motion of the moving body between the first time and the second time. ### Claim 4 (depends on 2) The method of claim 2, wherein applying the translation and rotation operations comprises applying a translate-rotate matrix to coordinates of pixel locations of the first image data. ### Claim 5 (depends on 4) The method of claim 4, wherein the translate-rotate matrix is: [ k ⁢ ⁢ Δ ⁢ ⁢ ψ n cos ⁢ ⁢ Δ ⁢ ⁢ ϕ n - sin ⁢ ⁢ Δ ⁢ ⁢ ϕ n k ⁢ ⁢ Δ ⁢ ⁢ θ n sin ⁢ ⁢ Δ ⁢ ⁢ ϕ n cos ⁢ ⁢ Δ ⁢ ⁢ ϕ n ] wherein k is a number of pixels per degree of a field of view of the imager device; wherein Δψn is an angular displacement sensed by the IMU between the first time and the second time along an axis corresponding to a yaw axis of the moving body; wherein Δθn is an angular displacement sensed by the IMU between the first time and the second time along an axis corresponding to a pitch axis of the moving body; and wherein Δϕn is an angular displacement sensed by the IMU between the first time and the second time along an axis corresponding to a roll axis of the moving body. ### Claim 6 (depends on 2) The method of claim 2, further comprising: resampling the registered first image data subsequent to applying the translation and rotation operations to the coordinates of the pixel locations. ### Claim 7 (depends on 1) The method of claim 1, wherein receiving the inertial measurement data sensed by the IMU comprises receiving angular rate data sensed by the IMU along three mutually-orthogonal axes. ### Claim 8 (depends on 7) The method of claim 7, wherein registering the first image data to the second image data based on the received inertial measurement data comprises: integrating the sensed angular rate data from the first time to the second time to produce angular displacement data; and registering the first image data to the second image data based on the angular displacement data. ### Claim 9 (depends on 8) The method of claim 8, wherein registering the first registered image data to the second image data based on the identified features that are common to both the first registered image data and the second image data comprises determining image-based angular displacement data, the method further comprising: determining angular rate correction data based on the angular displacement data and the image-based angular displacement data; wherein integrating the sensed angular rate data from the first time to the second time to produce the angular displacement data comprises applying the angular rate correction data to the sensed angular rate data. ### Claim 10 (depends on 1) The method of claim 1, wherein the imager device has a field of view centered about a central boresight; wherein the IMU includes a plurality of rate gyroscopes, each of the plurality of rate gyroscopes configured to sense rotational rate of the IMU along one of a plurality of mutually-orthogonal axes; and wherein one of the plurality of mutually-orthogonal axes is aligned with the central boresight of the imager device. ### Claim 11 (depends on 1) The method of claim 1, further comprising: determining, based on pixel locations of the identified target within the second image data and the output registered image data, that the identified target is moving; wherein modifying the trajectory of the moving body based on the line-of-sight vector comprises modifying the trajectory of the moving body to intercept the identified target. ### Claim 12 A system comprising: an imager device disposed on a moving body; an inertial measurement unit (IMU) disposed on the moving body; one or more processors; and computer-readable memory encoded with instructions that, when executed by the one or more processors, cause the system to: receive first image data that is captured at a first time by the imager device; receive second image data that is captured by the imager device at a second time that is subsequent to the first time; receive inertial measurement data sensed by the IMU, the inertial measurement data corresponding to sensed motion of the moving body between the first time and the second time; register the first image data to the second image data based on the received inertial measurement data to produce first registered image data; in response to identifying that image features are common to both the first registered image data and the second image data: register the first registered image data to the second image data based on the identified features that are common to both the first registered image data and the second image data to produce second registered image data; and output the second registered image data as output registered image data; and output the first registered image data as the output registered image data in response to determining that no image features are common to both the first registered image data and the second image data; identify a target within the second image data and the output registered image data; determine a line-of-sight vector from the imager device to the target based on pixel locations of the identified target within the second image data and the output registered image data; and output guidance and control commands configured to modify a trajectory of the moving body based on the line-of-sight vector. ### Claim 13 (depends on 12) The system of claim 12, wherein the computer-readable memory is further encoded with instructions that, when executed by the one or more processors, cause the system to register the first image data to the second image data based on the received inertial measurement data by at least causing the system to apply translation and rotation operations to pixel locations of the first image data based on the received inertial measurement data. ### Claim 14 (depends on 13) The system of claim 13, wherein the translation operations correspond to pitching and yawing motion of the moving body between the first time and the second time; and wherein the rotation operations correspond to rolling motion of the moving body between the first time and the second time. ### Claim 15 (depends on 12) The system of claim 12, wherein the imager device has a field of view centered about a central boresight; wherein the IMU includes a plurality of rate gyroscopes, each of the plurality of rate gyroscopes configured to sense rotational rate of the IMU along one of a plurality of mutually-orthogonal axes; and wherein one of the plurality of mutually-orthogonal axes is aligned with the central boresight of the imager device. ### Claim 16 (depends on 15) The system of claim 15, wherein the computer-readable memory is further encoded with instructions that, when executed by the one or more processors, cause the system to register the first image data to the second image data based on the received inertial measurement data by at least causing the system to: integrate the sensed rotational rate of the IMU from the first time to the second time to produce angular displacement data; and register the first image data to the second image data based on the angular displacement data. ### Claim 17 (depends on 12) The system of claim 12, wherein the computer-readable memory is further encoded with instructions that, when executed by the one or more processors, cause the system to: determine, based on pixel locations of the identified target within the second image data and the output registered image data, that the identified target is moving; and output the guidance and control commands configured to modify the trajectory of the moving body based on the line-of-sight vector to intercept the identified target.