Method and apparatus for estimating scene structure and ego-motion from multiple images of a scene using correlation

Apparatus for estimating both three-dimensional (3D) scene structure and ego-motion from a batch of images of the scene comprising:

• at least one camera which obtains a batch of images including a reference image and a plurality of inspection images as the at least one camera moves through the scene;
• means for providing an initial estimate of the ego-motion and the scene structure for the batch of images;
• a correlation processor, responsive to the initial estimate of ego-motion and scene structure, 1) to globally correlate each of the inspection images to the reference image, the correlation processor defining a global ego-motion constraint for all of the inspection images relative to the reference image and 2) to locally correlate each of the inspection images to the reference image to define a plurality of local structure constraints for all of the inspection images relative to the reference image;
• a processor, coupled to the correlation processor to define a differential ego-motion estimate from the global ego-motion constraint and to define a differential structure estimate from the plurality of local structure constraints;
• a plurality of adders which add the differential ego-motion estimate to the initial ego-motion estimate to provide a refined ego-motion estimate and which add the differential structure estimate to the initial structure estimate to provide a refined structure estimate.

An article of manufacture comprising a carrier containing computer program instructions, the computer program instructions controlling a general purpose computer to estimate both three-dimensional (3D) scene structure and ego-motion from a batch of images of the scene obtained by a camera as it moves through the scene, the computer program instructions causing the computer to perform the steps of:

• defining a reference image and a plurality of inspection images in the batch of images;
• providing an initial estimate of the ego-motion and the scene structure for the batch of images;
• responsive to the initial estimate of ego-motion and scene structure, globally correlating each of the inspection images to the reference image to define a global ego-motion constraint for all of the inspection images relative to the reference image;
• refining the initial estimate of ego-motion based on the global ego-motion constraint;
• responsive to the initial estimate of ego-motion and scene structure, locally correlating each of the inspection images to the reference image to define a plurality of local structure constraints for all of the inspection images relative to the reference image;
• responsive to the plurality of local structure constraints, refining the initial estimate of scene structure in respective regions of the reference image corresponding to the plurality of local structure constraints.

A method of generating a depth map of a scene from a sequence of images of the scene obtained by a camera moving through the scene, the method comprising the steps of:

• a) selecting a first batch of images from the sequence of images;
• b) processing the first batch of images to generate estimates of ego-motion of the camera and structure of the scene;
• c) projecting the estimated structure of the first batch of images into a world coordinate system;
• d) selecting a further batch of images from the sequence of images, the further batch of images having a reference image that is included in a previous batch of images;
• e) using the estimated ego-motion for the previous batch of images, mapping the structure estimate for the previous batch of images into a coordinate system defined by the reference image of the further batch of images;
• f) processing the further batch of images to generate further estimates of ego-motion of the camera and structure of the scene;
• g) projecting the further estimated structure into the world coordinate system to be combined with the previously projected estimated structure;
• h) repeating steps d) through g) until a last batch of images in the sequence of images has been processed;
• and
• i) providing the combined projected estimated structure as the depth map.