# US 2003/0202701 A1 · Method and apparatus for tie-point registration of disparate imaging sensors by matching optical flow

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2003/0202701 A1
- Document ID: US20030202701A1
- Status: Published application
- Kind: A1
- Application number: 10/113,641
- Filed: 2002-03-29
- Published: 2003-10-30
- Earliest claimed: 2002-03-29
- Inventors: Jonathon Schuler
- Web page: https://patentgrep.com/patent/US20030202701A1

## CPC

- G06T 7/33: using feature-based methods
- G06T 7/38: Registration of image sequences
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus for enabling the registration of co-located, disparate imaging sensors by computing the optical flow of each sensor as all the sensors simultaneously observe a moving object, or as all the sensors simultaneously move observing an object. The tie point registration of disparate imaging sensors is made more robust by matching optical flow and by levering the temporal motion within a pair of video sequences and using an additional constraint to minimize the disparity in optical flow between registered video sequences. The method includes parametrically computing the optical flow of each video sequence separately relative to a reference frame pair, identifying a matching constellation of tie-points in the reference pair of images, for all frames, computing the positions of tie-points bi=b0+ei where ei=predictive term to generate a new set of tie-points, after transformation by optical flow. For each frame, the total squared error resulting from an over-determined solution of affine registration problem is computed. The choice of ei is adjusted to minimize the total squared error over all frames of video.

## Claims

### Claim 1

A method for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, the method comprising:
computing a set of parametric flow field matrices for at least two video sequences having consecutive image pairs for each separate imaging sensor;
determining a set of feature tie points for a reference frame pair;
evaluating locations of the feature tie points for all subsequent image pairs; and
redefining the tie point registration as simultaneous least squares solution to aligning all candidate tie points.

### Claim 2 (depends on 1)

The method as in claim 1, wherein the set of parametric flow field matrices are defined as {FiA, FiB} i=1,2, . . . , n−1 relative to a video frame pair {A0, B0}.

### Claim 3 (depends on 1)

The method as in claim 1, wherein the locations of the feature tie points for all subsequent image pairs is evaluated by the relation:
ai=a0+└1 a0┘FiAi=1,2, . . . , n−1bi=b0+[1 b0]FiB
where
a0=tie points in frame A0
b0=tie points in frame B0.

### Claim 4 (depends on 1)

The method as in claim 1, wherein the imaging sensors are electro-optic imaging sensors.

### Claim 5 (depends on 1)

The method as in claim 1, wherein the imaging sensors comprise a visible imager and an infrared imager.

### Claim 6

A method for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, the method comprising:
identifying multiple pairs of frames in a video sequence;
computing the optical flow of a plurality of video sequences;
computing positions of tie points across the plurality of video sequences; and
if one of the tie points has an initial error, adjusting the initial error such that error over all optical flow tie points is less than the initial error.

### Claim 7 (depends on 6)

The method as in claim 6, wherein frame-to-frame estimation of a video scene motion of an individual imaging sensor is computed using shift estimation techniques.

### Claim 8 (depends on 6)

The method as in claim 6, wherein motion of video sequences is characterized as flow field parameterized by geometry of imaging sensor motion and spatial distortions of imaging sensor optics.

### Claim 9

A method for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, the method comprising:
identifying a reference set of tie points b0;
seeking a corrective term e0 to generate a new set of tie points a′=a0+e0, such that the corrective term minimizes fitted registration error of tie points in all frames of a video sample.

### Claim 10 (depends on 9)

The method as in claim 9, further comprising the steps of:
parametrically computing optical flow of each separate video sequence relative to a reference frame pair;
selecting a matching constellation of tie points in the reference frame pair;
for all frames, computing the positions of tie points b0 and ai=a0+ei after transformation by optical flow;
for each frame, computing the total squared error from an over-determined solution of affiance registration; and
adjusting the choice of ei to minimize the total squared error over all frames of video to improve the accuracy of tie point registration of disparate imaging sensors.

### Claim 11 (depends on 9)

The method as in claim 9, wherein the reference set of tie points b0 is absolute.

### Claim 12

A method for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, the method comprising:
identifying an initial set of tie points to define a registration model to align a first image A onto a second base image B;
defining total registration error of the first and second images as a function of {a0, b0, FiA, FiB} where i=1, . . . (N−1); and
adjusting one set of tie points a0 so as to minimize registration error such that error of {a0′, b0, FiA, FiB}<Error of {a0, b0, FiA, FiB}, where a0′=a0+e0 and e0 is a corrective term to generate a new set of tie points.

### Claim 13

A method for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, the method comprising:
given tie-points in image B as absolute, compute {B0, B1, . . . Bn} where B1 . . . Bn represent flow estimates of a first image, and B0 represents an original tie-point;
given A+ε0 tie-points, compute {A0, A1, . . . An} where A1, . . . An represents flow estimates from data of a second image and ε0 represents a corrective term; and
adaptively choose ε0 to minimize error between {A0, A1, . . . An} and {B0, B1, . . . Bn}.

### Claim 14

An apparatus for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, comprising:
means for identifying a reference set of tie points b0;
means for seeking a corrective term e0 to generate a new set of tie points a′=a0+e0, such that the corrective term minimizes fitted registration error of tie points in all frames of a video sample.

### Claim 15 (depends on 14)

The apparatus as in claim 14, further comprising:
means for parametrically computing optical flow of each separate video sequence relative to a reference frame pair;
means for selecting a matching constellation of tie points in the reference frame pair;
means for computing the positions of tie points b0 and a1=a0+ei after transformation by optical flow;
means for computing the total squared error from an over-determined solution of affiance registration; and
means for adjusting the choice of ei to minimize the total squared error over all frames of video to improve the accuracy of tie point registration of disparate imaging sensors.

### Claim 16 (depends on 14)

The apparatus as in claim 14, wherein the reference set of tie points b0 is absolute.

### Claim 17

An apparatus for improving the accuracy of tie point registration of disparate imaging sensors by matching optical flow, comprising:
means for identifying multiple pairs of frames in a video sequence;
means for computing the optical flow of a plurality of video sequences;
means for computing positions of tie points across the plurality of video sequences; and
means for adjusting an initial error, if one of the tie points has the initial error, such that error over all optical flow tie points is less than the initial error.

### Claim 18 (depends on 17)

The apparatus method as in claim 17, wherein frame-to-frame estimation of a video scene motion of an individual imaging sensor is computed using shift estimation techniques.

### Claim 19 (depends on 17)

The apparatus as in claim 17, wherein motion of video sequences is characterized as flow field parameterized by geometry of imaging sensor motion and spatial distortions of imaging sensor optics.
