# US 6,833,843 B2 · Panoramic imaging and display system with canonical magnifier

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 6,833,843 B2
- Document ID: US6833843B2
- Status: Granted
- Kind: B2
- Application number: 10/005,824
- Filed: 2001-12-03
- Published: 2003-06-05
- Granted: 2004-12-21
- Earliest claimed: 2001-12-03
- Assignees: Tempest Microsystems
- Inventors: Michael Mojaver; Steven Branson
- Web page: https://patentgrep.com/patent/US6833843B2

## CPC

- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 3/00: Geometric image transformations in the plane of the image
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 23/58: Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present invention provides an image acquisition and viewing system that employs a fish-eye lens and an imager, such as, a CCD array, to obtain a wide angle image, e.g., an image of a hemispherical field of view. The system of the invention further includes a primary display for displaying the image, and a secondary display for presenting a perspective-corrected view of a selected portion of the image. A processor effects correction of a selected portion of the image for distortions introduced by the fish-eye lens by utilizing methods according to the teachings of the invention. The processor further effects the display of the image and its perspective-corrected portion thereof on the primary and the secondary displays, respectively. Moreover, the processor effects the display of a graphical object on the first display to delineate the portion that is presented on the secondary display. This allows a viewer to view simultaneously the image and the perspective-corrected portion, thereby gleaning information regarding the context of the perspective-corrected portion within the entire image while viewing it.

## Claims

### Claim 1

A device for providing a perspective-corrected view of at least a portion of a wide angle image, comprising:
an image-capturing device having a fish-eye lens for generating the wide angle image as a projection of at least a section of a viewing hemisphere associated with the lens onto an image plane,
a processor coupled to the image capturing device that receives the image, the processor further comprising:
a first display module for displaying the wide angle image, and
a second display module for displaying a perspective-corrected view of portion of the image,
the processor further comprising a correlation module for displaying a graphical object on the displayed wide angle image that identifies the selected portion of the image displayed in the perspective-corrected view,
wherein the processor generates said perspective-corrected view from a vantage point offset relative to a center of said viewing hemisphere.

### Claim 2 (depends on 1)

The device of claim 1, wherein the processor further comprises a scale adjuster that applies to said perspective-corrected view a user-adjustable magnification.

### Claim 3 (depends on 2)

The device of claim 2, wherein the processor generates the perspective-corrected view of a portion of the image by transforming that portion according to a defined magnification and an angle for viewing a section of the hemisphere corresponding to that image portion from said offset vantage point.

### Claim 4 (depends on 1)

The device of claim 1, further comprising a user interface module coupled to the processor for selecting a portion of the image to be displayed as a perspective corrected view.

### Claim 5 (depends on 4)

The device of claim 4, wherein the user interface allows selecting a magnification for viewing the perspective-corrected portion of the image.

### Claim 6 (depends on 1)

The device of claim 1, wherein the wide angle image is a fish-eye image.

### Claim 7 (depends on 6)

The device of claim 6, wherein the image-capturing device further comprises an imager coupled to the fish-eye lens for converting optical photons collected by the lens from the field of view into electrical signals representing an image of the field of view.

### Claim 8 (depends on 7)

The device of claim 7, wherein the imager can be any of a CCD array, a CMOS array, or a thermal imaging device.

### Claim 9 (depends on 7)

The device of claim 7, wherein the imager has a resolution in a range of about one million to about 100 million effective pixels per square inch.

### Claim 10 (depends on 6)

The device of claim 6, wherein the processor generates the perspective-corrected view by mapping a point (u,v) on an undistorted image plane corresponding to a perspective-corrected portion of a distorted image to a point (x,y) on a plane corresponding to the distorted image, the mapping is provided by the following equations:
x=R(β0/(PI/2))cos(d0)
y=R(β0/(PI/2))sin(d0)
t=[Dd+sqrt(D2d2−(u2+v2+d2)(D2−1))]/(u2+v2+d2)
β0=arctan(−D cos(β)+dt cos(β)−vt sin(β), 1)
δ0=arctan(−D sin(β)+dt sin(β)+vt cos(β), ut)+δ
where β and δ are the zenith and azimuthal angles corresponding to the center of the undistorted image, β0 and δ0 are the zenith and azimuthal angles corresponding to a pixel (u,v) in the undistorted image, d is a magnification factor, D is a level of undistortion, and R is the radius of the fisheye image.

### Claim 11 (depends on 10)

The device of claim 10, wherein each of the distorted and undistorted image planes includes a two-dimensional array of pixels providing a digital luminance map corresponding to the image or a perspective-corrected portion of the image, respectively.

### Claim 12 (depends on 11)

The device of claim 11, wherein the processor generates the luminance value of a pixel corresponding to the point (u,v) by calculating a weighted average of luminance values of two or more pixels surrounding the point (x,y), if the point (x,y) does not correspond to a pixel on the distorted image plane.

### Claim 13

A device for providing a perspective-corrected view of at least a portion of a wide angle image, comprising:
an image-capturing device having a fish-eye lens for generating the wide angle image,
a processor in communication with the image-capturing device,
the processor correcting at least a portion of the image for distortions introduced by the fish-eye lens by mapping a point (u,v) on an undistorted image plane corresponding to a perspective-corrected view of the image portion to a point (x,y) on a distorted image plane corresponding to the image in accord with an angle for viewing a section of the hemisphere corresponding to the image portion from a vantage point offset from a center of the hemisphere and a distance between the vantage point and the center of the hemisphere.

### Claim 14 (depends on 13)

The device of claim 13, wherein the processor employs the following equations to effect the mapping between the point (u,v) and the point (x,y):
x=R(β0/(PI/2))cos(d0)
y=R(β0/(PI/2))sin(d0)
t=[Dd+sqrt(D2d2−(u2+v2+d2)(D2−1))]/(u2+v2+d2)
β0=arctan(−D cos(β)+dt cos(β)−vt sin(β), 1)
δ0=arctan(−D sin(β)+dt sin(β)+vt cos(β), ut)+δ
where β and δ are the zenith and azimuthal angles corresponding to the center of the undistorted image, β0 and δ0 are the zenith and azimuthal angles corresponding to a pixel (u,v) in the undistorted image, d is the magnification factor, D is the level of undistortion, and R is the radius of the fisheye image.

### Claim 15

A method for generating a perspective-corrected view of a portion of a fish-eye image, comprising:
mapping a point (u,v) on an undistorted image plane corresponding to a perspective view of the image portion to a point (x,y) on a distorted image plane corresponding to the fish-eye image according to the following equations:
x=R(β0/(PI/2))cos(d0)
y=R(β0/(PI/2))sin(d0)
t=[Dd+sqrt(D2d2−(u2+v2+d2)(D2−1))]/(u2+v2+d2)
β0=arctan(−D cos(β)+dt cos(β)−vt sin(β), 1)
δ0=arctan(−D sin(β)+dt sin(β)+vt cos(β), ut)+δ
where β and δ are the zenith and azimuthal angles corresponding to the center of the undistorted image, β0 and δ0 are the zenith and azimuthal angles corresponding to a pixel (u,v) in the undistorted image, d is the magnification factor, D is the level of undistortion, and R is the radius of the fisheye image.