System for executing 3D propagation for depth image-based rendering

A method for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera by a computer having a processor and memory, the method comprising:

• propagating, by the computer, depth information from the depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, where propagating includes 3D warping with the depth camera considered as a first reference view and each color camera respectively being considered as a first target view, where the one or more color cameras and the at least one depth camera are positioned arbitrarily at different locations relative to a scene to be rendered;
• enhancing, by the computer, the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images by executing the following processing techniques:
• detect occluded pixels in the propagated depth images;
• occlusion removal to replace values of the occluded pixels with newly interpolated values;
• depth-color bilateral filtering (DCBF) to combine color and propagated depth information to calculate values for unknown depth pixels in the propagated depth image while preserving edges of an object in the scene;
• directional disocclusion filling to fill areas of unknown depth pixels caused by the disocclusion in the propagated depth image, the filling direction originating from a plurality of epipoles related to each color camera;
• and
• depth edge enhancement to sharpen depth edges surrounding objects in the propagated depth images comprising:
• computing depth gradients in vertical, horizontal, and in at least one diagonal direction with Sobel operators;
• for each pixel P whose depth edge gradients are greater than a predetermined value, searching within a search window among the neighbor pixels for a best color-matched pixel that has a smallest Euclidean color distance with pixel P;
• and
• replacing the depth value of the pixel P by the depth value of the best color-matched pixel;
• and
• rendering, by the computer, a complete image viewable on a screen of the computer from one or more enhanced depth images from the one or more color cameras.