Back to results
US 6,636,234 B2Granted
Image processing apparatus for interpolating and generating images from an arbitrary view point
AssigneeCanon Kabushiki Kaisha
InventorsTakaaki Endo, Akihiro Katayama
Life of this patent
Estimating the term…
Abstract
There is provided a method of generating an interpolated image without any constraint even in a wide moving range. When two real images have an inclusive relationship, the two images are rotated to make their optical axes coincide with each other. Nonlinear interpolation processing is performed for the two images after the rotations, and the interpolated image is rotated in the line of sight direction of a user.
Claims67
A method of generating an interpolated image at a predetermined viewpoint position on the basis of first and second images obtained at first and second viewpoint positions different from the predetermined viewpoint position, comprising:
- an inputting step of inputting the first and second images;
- a determination step of determining whether a field of view of the first image includes a lens center of the second image;
- and
- an image generation step of generating an interpolated image by performing steps (A) to (C) when the field of view of the first image includes the lens center of the second image in said determination step and performing steps (D) to (F) when the field of view of the first image does not include the lens center of the second image in said determination step;
- (A) a first correcting step of correcting the first and second images, respectively, such that optical axes of cameras which respectively sensed the first and second images coincide with a first direction substantially coinciding with a direction of a base line connecting the first and second viewpoint positions;
- (B) a first interpolation step of calculating coordinates of a corresponding point of an interpolated image having the first direction as a line of sight direction at the predetermined viewpoint position by performing nonlinear proportional distribution for coordinates of a first corresponding point in the first image with respect to a predetermined feature point and coordinates of a second corresponding point in the second image with respect to the feature point;
- (C) a second correcting step of correcting the interpolated image from the first direction to the line of sight direction at the predetermined viewpoint position;
- (D) a third correcting step of correcting the first and second images, respectively, such that optical axes of the image become substantially perpendicular to the direction of the base line;
- (E) a second interpolation step of calculating coordinates of a corresponding point of an interpolated image having the second direction perpendicular to the direction of the base line at the predetermined viewpoint position as a line of sight direction by performing linear proportional distribution for coordinates of a first corresponding point of the first image and coordinates of a second corresponding point of the second image;
- (F) a fourth correcting step of correcting the interpolated image from the second direction to the line of sight direction at the predetermined viewpoint position, and outputting the image after the fourth correction as an interpolated image.
7 more elements
A method of generating an interpolated image at a predetermined viewpoint position on the basis of first, second, and third images respectively obtained at first, second, and third viewpoint positions different from the predetermined viewpoint position, comprising:
- an inputting step of inputting the first, second and third images;
- an internal ratio calculation step of obtaining first and second internal ratios on the basis of a relationship between the predetermined viewpoint position and the first to third viewpoint positions;
- a first interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;
- a second interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;
- a first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of the first interpolation step and the second interpolation step for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship determined by determining if a field of view of one of the first image and the second image includes a lens center of the other;
- a second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first interpolation step and the second interpolation step for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship determined by determining if a field of view of one of the temporary virtual image and the third image includes a lens center of the other;
- a rotation step of rotating the interpolated image, obtained in said second image interpolation step, in a line of sight direction at the predetermined viewpoint position;
- and
- an outputting step of outputting the image after the rotation as an interpolated image.
6 more elements
An image processing apparatus comprising:
- image storing means for storing a plurality of images together with viewpoint position information corresponding thereto;
- detecting means for detecting both a line of sight direction and viewpoint position of a user;
- searching means for searching, in said image storing means for the first to third images including corresponding points in the line of sight direction of the user;
- calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;
- first interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;
- second interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;
- first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of the nonlinear proportional distribution processing and the linear proportional distribution processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship determined by determining if a field of view of one of the first image and the second image includes a lens center of the other;
- second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the nonlinear proportional distribution processing and the linear proportional distribution processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship determined by determining if a field of view of one of the temporary virtual image and the third image includes a lens center of the other,
- image rotation means for rotating the interpolated image, obtained in the second image interpolation means, in a line of sight direction at the user viewpoint position;
- and
- outputting means for outputting the image after the rotation as an interpolated image.
8 more elements
An image processing apparatus for generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions, comprising:
- segmentation means for segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position;
- and
- processing means for performing a morphing process on the first and second images on the basis of the depths and a change of a viewpoint position.
An image processing method of generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions to obtain a morphed image comprising:
- a segmentation step of segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position;
- and
- a processing step of performing a morphing process on the first and second images on the basis of the depths and a change of a viewpoint position.
An image processing apparatus for interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions, comprising:
- discrimination means for discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images;
- and
- image processing means for interpolating the plurality of images on the basis of a discrimination result obtained by said discrimination means,
- wherein said image processing means is designed to use different interpolation methods depending on whether said discrimination means discriminates the presence or absence of an inclusive relationship between the plurality of images.
1 more element
An image processing method of interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions, comprising:
- a discrimination step of discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images;
- and
- a image processing step of interpolating the plurality of images on the basis of a discrimination result obtained in said discrimination step,
- wherein said image processing step comprises using different interpolation methods depending on whether the presence or absence of an inclusive relationship between the plurality of images is discriminated in said discrimination step.
1 more element
Dependencies are inferred from the claim text.