# US 7,412,091 B2 · System and method for registration of cubic fisheye hemispherical images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,412,091 B2
- Document ID: US7412091B2
- Status: Granted
- Kind: B2
- Application number: 10/433,906
- Filed: 2001-12-07
- Published: 2004-04-01
- Granted: 2008-08-12
- Earliest claimed: 2000-12-07
- Assignees: iLOOKAbout Inc.
- Inventors: Jeff Hack
- Web page: https://patentgrep.com/patent/US7412091B2

## CPC

- G06T 3/12: Panospheric to cylindrical image transformations
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for registering hemispheric images obtained using a fisheye lens for panoramic viewing, relating to spatial alignment and colour balancing of complement half-side images derived from the hemispheric images. The systems and methods for spatial alignment determine the displacement of the digitized recorded images using a single translation and rotation model of distortion, which further evaluates the centre of projection, and the extent of translation and rotation. The system and method for colour balancing iteratively increase (or decrease) the values of pixels near the edge (discontinuity) between the two half-sides of an image, each time taking into consideration the average difference of pixel colour across a smaller strip of the edge near the pixel. This invention removes the colour differential but does not remove any detail underlying the features of the image.

## Claims

### Claim 1

A computer-assisted method for aligning two digitized overlapping hemispheric images of an environment obtained with a fisheye lens comprising the steps of:
(a) receiving the location of three spatial points as recorded in an overlapping region of each hemispheric image;
(b) determining the translation and rotation of one hemispheric image relative to its ideal position from the locations of the three spatial points in the two images based on a displacement model; and
(c) obtaining the true centre of projection (Q) in each image.

### Claim 2

A system for aligning two digitized overlapping hemispheric images of an environment obtained with a fisheye lens comprising:
(a) a unit for receiving the location of three spatial points as recorded in an overlapping region of each hemispheric image;
(b) a unit for determining the translation and rotation of one hemispheric image relative to its ideal position from the locations of the three spatial points in the two images based on a displacement model; and
(c) a unit for obtaining the true centre of projection (Q) in each image.

### Claim 3

A method for balancing the color difference of two pixelated images joined together at an edge, comprising iteratively increasing (or decreasing) the values of pixels near the edge, each time taking into consideration the average difference of pixel colour across a smaller strip of the edge near the pixel.

### Claim 4 (depends on 1)

The method of claim 1 where the spatial points comprise pairs of complementary points, with each complementary point of the pair correlating to a different hemispheric image, and where the one hemispheric image has a centre of projection (q).

### Claim 5 (depends on 4)

The method of claim 4 where each complementary point of the pair corresponds to the same common point on each hemispheric image.

### Claim 6 (depends on 4)

The method of claim 4 where the displacement model comprises aligning the complementary points by making the zenith angle φ for the complementary points of each pair equal.

### Claim 7 (depends on 4)

The method of claim 4 where aligning comprises:
forming complementary vectors between each complementary point from both hemispheric images and the centre of projection (q); and
displacing the one hemispheric image from the other hemispheric image, through rotations and translations of the vectors, so that the complementary vectors are parallel and pass through the centre of projection (q).

### Claim 8 (depends on 7)

The method of claim 7 where displacing comprises making the vector cross product of the vectors zero.

### Claim 9 (depends on 7)

The method of claim 7 further comprising determining the number of pixels equal to 180 degrees of view.

### Claim 10 (depends on 9)

The method of claim 9 where determining the number of pixels equal to 180 degrees of view comprises computing, for each pair of complementary points, the average of a radial distance between the centre of projection (q) and each point, after the one hemispheric image has been displaced.

### Claim 11 (depends on 9)

The method of claim 9 further comprising generating six perspective sides of a cubic environmental map using the true centre of projection (Q) of each hemispherical image and the number of pixels equal to 180 degrees of view.

### Claim 12 (depends on 1)

The method of claim 1 where the three spatial points are non-collinear.

### Claim 13 (depends on 5)

The method of claim 5 where receiving the location of the common points comprises manually selecting the complementary points by selecting corresponding features in each hemispherical image.

### Claim 14 (depends on 5)

The method of claim 5 where receiving the location of the common points comprises automatically selecting the complementary points by selecting corresponding features in each hemispherical image.

### Claim 15 (depends on 5)

The method of claim 5 where the overlapping region is an annulus, and receiving the location comprises selecting the common points are from different thirds of the annulus.

### Claim 16 (depends on 5)

The method of claim 5 where receiving the location comprises a user selecting a spatial point in the one hemispheric image and an approximate area in the other hemispheric image so that an electronic processor locates the precise spatial point in the second image.

### Claim 17 (depends on 2)

The system of claim 2 where the spatial points comprise pairs of complementary points, with each complementary point of the pair correlating to a hemispheric image, and where the one hemispheric image has a centre of projection (q).

### Claim 18 (depends on 17)

The system of claim 17 where each complementary point of the pair corresponds to the same common point on each hemispheric image.

### Claim 19 (depends on 17)

The system of claim 17 where the displacement model comprises aligning the complementary points by making the zenith angle φ for the complementary points of each pair equal.

### Claim 20 (depends on 17)

The system of claim 17 where aligning comprises:
forming vectors between each complementary point from both hemispheric images and the centre of projection (q); and
displacing the one hemispheric image from the other hemispheric image, through rotations and translations of the vectors, so that the complementary vectors are parallel and pass through the centre of projection (q).

### Claim 21 (depends on 20)

The system of claim 20 where displacing comprises making the vector cross product of the vectors zero.

### Claim 22 (depends on 20)

The system of claim 20 further comprising a unit for determining the number of pixels equal to 180 degrees of view.

### Claim 23 (depends on 22)

The system of claim 22 where determining the number of pixels equal to 180 degrees of view comprises finding the average of the radial distance for any pair of complementary points, after the one hemispheric image has been displaced.

### Claim 24 (depends on 22)

The system of claim 22 further comprising a unit for generating six perspective sides of a cubic environmental map using the true centre of projection (Q) of each hemispherical image and the number of pixels equal to 180 degrees of view.

### Claim 25 (depends on 2)

The system of claim 2 where the three spatial points are non-collinear.

### Claim 26 (depends on 18)

The system of claim 18 where the unit for receiving the location of the common points comprises a user interface for manually selecting the complementary points by selecting corresponding features in each hemispherical image.

### Claim 27 (depends on 18)

The system of claim 18 where the unit for receiving the location of the common points comprises an electronic processor for automatically selecting the complementary points by selecting corresponding features in each hemispherical image.

### Claim 28 (depends on 18)

The system of claim 18 where the overlapping region is an annulus, and the unit for receiving the location comprises a interface for selecting the common points are from different thirds of the annulus.

### Claim 29 (depends on 18)

The system of claim 18 where the unit for receiving the location comprises a user interface for selecting a spatial point in the one hemispheric image and an approximate area in the other hemispheric image so that an electronic processor locates the precise spatial point in the second image.

### Claim 30 (depends on 3)

The method of claim 3, wherein the values are colour values.

### Claim 31 (depends on 3)

The method of claim 3, wherein the values are increased or decreased by a variation term, which is a function of the average difference across the edge.

### Claim 32 (depends on 3)

The method of claim 3, wherein the strip comprises pixels from each of the two images.

### Claim 33 (depends on 3)

The method of claim 3, wherein the pixels near the edge comprise a column of pixels perpendicular to the edge at an index, i, defined by a pair of adjacent pixels across the edge.

### Claim 34 (depends on 33)

The method of claim 33, wherein the strip comprises pixels from each of the two images and wherein the difference is calculated between pixels from each image that are adjacent to one another across the edge.

### Claim 35 (depends on 34)

The method of claim 34, wherein the strip is centered at the index, i.

### Claim 36 (depends on 33)

The method of claim 33, wherein the value of each pixel in the column is increased or decreased by a variation term, which is weighted decreasingly with the distance of each pixel from the edge.

### Claim 37 (depends on 33)

The method of claim 33, wherein the strip has a width along the edge equal to about double a length, w, of the column.

### Claim 38 (depends on 37)

The method of claim 37, wherein, upon each iteration, the length, w, decreases.

### Claim 39 (depends on 33)

The method of claim 33, wherein the method is repeated for a plurality of the index, i, along the edge.

### Claim 40 (depends on 39)

The method of claim 39, wherein the plurality comprises each pair of adjacent pixels along the edge.