US 5,703,961 A · Image transformation and synthesis methods

Abstract

A system for generating images of a scene as the scene would be observed from an arbitrary location. A plurality of discrete images, typically video images, taken at different viewpoints, as, for example, by a plurality of cameras pointing outwardly on a curving locus are converted to an offset epipolar image. The offset epipolar image includes a plurality of linesets, each such lineset incorporating one scanning line from each of the discrete video images. Each line in the virtual image is reconstructed from a lineset of the epipolar image. The reconstruction may include interpolation between pixel data representing lines from adjacent discrete images and mapping of pixels from one or more lines representing one or more adjacent discrete images onto the pixel line of the virtual image. The nature of the mapping depends upon the viewpoint selected for the virtual image. The system can provide real time stereoscopic telepresence, i.e., a virtual viewpoint images for each eye as the observer moves his or her head.

Claims50

Dependent claims

A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint comprising the steps of:

(a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewpoints, each said discrete image including an array of pixel data in a first image dimension corresponding to position of depicted objects in a first dimension of said scene and in a second image dimension orthogonal thereto;
(b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each one of said line sets including one line of pixel data in said first image dimension from each one of said discrete images, all of the lines in each one of said line sets corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to an order of positions of said discrete viewpoints in said first dimension of said scene, each pair of adjacent lines of pixel data within each line set being offset from one another by an initial offset;
(c) providing pixel data for said synthetic image as a plurality of virtual viewpoint pixel lines extending in said first image dimension and displaced from one another in said second image dimension by (i) associating each virtual viewpoint line with a line set in said first epipolar image formed from lines having location in said second image dimension corresponding to the location in said second image dimension of that virtual viewpoint line and (ii) for each pixel within each virtual viewpoint line, deriving synthetic pixel data from pixel data in the associated line set.

A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location and viewing direction and having a predetermined field of view, the method comprising the steps of:

(a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewing directions from a plurality of discrete viewpoints on a predetermined viewpoint locus, said viewing directions being disposed at different angles relative to a reference line in a first dimension of said scene, each one of said discrete images including an array of pixel data in a first image dimension and in a second image dimension orthogonal thereto, so that position of each pixel in said first image dimension within each image will represent an angle between the viewing direction of the image and a ray direction from said pixel through the viewpoint of the image to a point in the scene represented by that pixel;
(b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to the order of said viewing directions in said first dimension of said scene, whereby each said line set defines an epipolar plane having first and second epipolar coordinates, each pixel in each line set having a location in said first epipolar coordinate and a location in said second epipolar coordinate, the location of each pixel in said second epipolar coordinate representing the viewing direction of the discrete image from which the line of pixel data incorporating that pixel was taken, the location of each pixel in said first epipolar coordinate representing a ray azimuth angle, the ray azimuth angle of each pixel being an angle in said first dimension of said scene between an index line and the ray direction of that pixel;
(c) selecting a base viewpoint on said viewpoint locus and a viewing direction associated with the base viewpoint, and;
(d) forming a line of the virtual viewpoint image from each said line set by:
(i) providing a base line within the line set having a second epipolar coordinate representing a viewing direction equal to the viewing direction associated with the base viewpoint;
(ii) mapping pixel data from the base line into the virtual viewpoint image line;
(iii) selecting supplementary pixels from one or more additional lines of the line set, said supplementary pixels having locations in said first and second epipolar coordinates close to the location of an end of the base line in said epipolar coordinates and
(iv) incorporating data from said supplementary pixels at an end of the virtual viewpoint image line corresponding to such end of the base line, steps (iii) and (iv) being performed at one end of the base line or at both ends of the base line so that the virtual viewpoint image line includes pixel data for said predetermined field of view.

5 more elements

A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location and viewing direction, the method comprising the steps of:

(a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewing directions from a plurality of discrete viewpoint locations on a predetermined viewpoint locus, said viewing directions being disposed at different angles relative to an index line in a first dimension of said scene, each said discrete image including an array of pixel data in a first image dimension and in a second image dimension orthogonal thereto, so that position of each pixel in said first image dimension within each image represents an angle between the viewing direction of the image and a ray direction for that pixel, the ray direction for each pixel being the direction of a ray from that pixel through the viewpoint of the image to a point in the scene represented by that pixel;
(b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to the order of said viewing directions, whereby each said line set defines an epipolar plane having first and second epipolar coordinates, each pixel in a line set having a location in the first and second epipolar coordinates of the epipolar plane defined by such line set, the location of each pixel in the second epipolar coordinate representing the viewing direction of the discrete image from which the line of pixel data incorporating that pixel was taken, the location of each pixel in said first epipolar coordinate representing a ray azimuth angle of that pixel, the ray azimuth angle of each pixel being an angle in said first dimension between said index line and the ray direction of that pixel;
(c) forming a line of the virtual viewpoint image from each said line set, each such line including a plurality of pixels each such pixel in the virtual viewpoint image having a view azimuth angle, the view azimuth angle of each such pixel being the angle between said index line and the direction of a ray from that pixel through the virtual viewpoint to a point in the scene represented by that pixel, the process of forming each such line in the virtual viewpoint image being performed by selecting a plurality of pixel sets in the virtual viewpoint line, each such pixel set including one pixel or a plurality of mutually adjacent pixels, each such pixel set defining a principal ray line extending through the virtual viewpoint location at a principal view azimuth angle close to the view azimuth angles of the pixels in such pixel set and, for each said pixel set:
(i) selecting one of said viewpoints on said locus as a base viewpoint such that an intercept of the principal ray line on the viewpoint locus is adjacent the base viewpoint and selecting a discrete image at the base viewpoint as a base image;
(ii) selecting a base line within the line set which has a location in the second epipolar coordinate equal to the location representing the viewing direction of the base image;
and
(iii) for each pixel in the set, selecting one or more pixels of the base line selected in step (ii) having locations in the first epipolar coordinate close to a location representing a ray azimuth angle equal to the view azimuth angle of the pixel in the set;
and mapping pixel data of the selected pixels of the base line into such pixel of the set.

5 more elements

A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location and virtual viewing direction, the method comprising the steps of:

(a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoint locations on a viewpoint locus having orthogonal first and second dimensions, each said discrete view direction defining a view direction vector, each said discrete image including pixel data for pixels in an array extending in a first image dimension and in a second image dimension orthogonal thereto, whereby position of each pixel within each image will represent differences in azimuth angle and elevation angle between the view direction vector of the discrete image and a ray direction vector from said pixel through the discrete viewpoint to a point in the scene represented by that pixel, the azimuth angle of a vector being the angle in said time dimension of said locus between such vector and an index line, the elevation angle of a vector being the angle in said second dimension of the locus between such vector and said index line;
(b) forming a two-dimensional array of virtual pixels constituting the synthetic image so that position of each virtual pixel in the array represents differences in azimuth and elevation angles between (i) a virtual ray direction vector from that virtual pixel through the virtual viewpoint to the point in the scene represented by that virtual pixel and (ii) a virtual view direction vector through the virtual viewpoint representing the virtual viewing direction;
(c) selecting a plurality of virtual pixel sets in the virtual array, each such virtual pixel set including one virtual pixel or a plurality of mutually adjacent virtual pixels, each said virtual pixel set defining a principal ray vector extending through the virtual viewpoint location in a principal view direction close to the virtual ray vector directions of the pixels in such pixel set and, for each said virtual pixel set:
(i) determining an intercept of the principal ray vector of that virtual pixel set on the viewpoint locus;
(ii) selecting one of said discrete images having a viewpoint location on said locus close to the intercept selected in step (i) as a base image for the virtual pixel set;
and
(iii) for each virtual pixel in the set, selecting one or more pixels of the base image selected in step (ii) having ray direction vectors with azimuth and elevation angles close to the azimuth and elevation angles of the virtual ray direction vector for that virtual pixel and mapping pixel data of the selected pixels of the base image into that virtual pixel.

4 more elements

(a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoints distributed in two orthogonal dimensions of said scene on a viewpoint locus, each said discrete view defining a view location vector having an azimuth component in an azimuth direction and an elevation component in an elevation direction orthogonal to the azimuth direction, each said discrete image including pixel data for pixels in an array extending in first and second image dimensions orthogonal to one another and to the view location vector, so that:
(i) each pixel in each said discrete image has a ray direction vector extending from such pixel through the viewpoint of the discrete image to a point in the scene depicted by such pixel, each such ray direction vector having azimuth and elevation components in said azimuth and elevation directions, respctively;
(ii) the position of each pixel in the first image direction corresponds to the difference between the azimuth component of the view location vector of the discrete image and the azimuth component of the ray direction vector of said pixel;
and
(iii) the position of each pixel in the second image direction corresponds to the difference between the elevation component of the view location vector of the discrete image and said ray direction vector of said pixel;
(b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered so that each said line set defines an epipolar space having a first epipolar coordinate corresponding to the azimuth components of the view location vectors, having a second epipolar coordinate corresponding to the elevation components of the view location vector and also corresponding to the elevation components of the ray direction vectors of the pixels, and having a third epipolar coordinate corresponding to the azimuth components of the ray direction vectors of the pixels and;
(c) forming a two-dimensional it array of virtual pixels constituting said synthetic image so that each said virtual pixel in the set has a ray direction vector from the virtual pixel through the virtual viewpoint location, each said ray direction vector from each virtual pixel having azimuth and elevation components in said azimuth and elevation directions, and
(d) selecting a plurality of pixel sets in said array of virtual pixels, each such pixel set including one virtual pixel or a plurality of mutually adjacent virtual pixels, each said pixel set defining a principal ray vector extending through the virtual viewpoint and having azimuth and elevation components close to the azimuth and elevation components of the ray direction vectors of the pixels in such pixel set and, for each said pixel set:
(i) determining an intercept of the principal ray vector on the viewpoint locus;
(ii) providing a line of one said line set having first and second epipolar coordinates corresponding to the azimuth and elevation components of a view location vector for a view location close to said intercept as a base line for the pixel set;
(iii) for each virtual pixel in the set, selecting one or more pixels of the base line having a third epipolar coordinate close to the azimuth component of the ray direction vector of that virtual pixel;
and
(iv) mapping pixel data of the selected pixels of the base image into such pixel of the pixel set.

10 more elements

A method of combining a first set of images of a first scene and a second set of images of a second scene, each said set of images including a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewpoints, each said discrete image including an array of pixel data arranged in a first image dimension corresponding to position of depicted objects in a first dimension in real space and in a second image dimension orthogonal thereto, the method including the steps of:

(a) constructing a first epipolar image from said first set of discrete images and a second epipolar image from said second set of discrete images, each said epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to an order of said viewpoints in said first real dimension;
and
(b) combining pixel data of each line set in said first epipolar image with pixel data of one line set in said second epipolar image to form combined line sets constituting a combined epipolar image.

Dependencies are inferred from the claim text.

Image transformation and synthesis methods

Life of this patent

Image transformation and synthesis methods

Life of this patent

Abstract

Claims50