Systems and Methods for Estimating and Refining Depth Maps
AssigneeFotoNation Limited
InventorsHossein Javidnia, Peter Corcoran
Life of this patent
Estimating the term…
Abstract
A method for improving accuracy of depth map information derived from image data descriptive of a scene. In one embodiment Mutual Feature Map data are created based on initial disparity map data values and the image data descriptive of the scene. The Mutual Feature Map data are applied to create a series of weighting functions representing structural details that can be transferred to the first disparity values to restore degraded features or replace some of the first disparity values with values more representative of structural features present in the image data descriptive of the scene.
Claims20
A method for improving accuracy of depth map information derived from image data descriptive of a scene where pixels of such image data, acquired with one or more image acquisition devices, each have an assigned intensity value, the method comprising:
- performing a matching cost optimization by iteratively refining disparities between corresponding pixels in the image data and using optimization results to create a sequence of first disparity values for an initial disparity map for the scene based in part on a superpixel-wise cost function;
- performing a guided filter operation on the first disparity values by applying other image data containing structural details that can be transferred to the first disparity values to restore degraded features or replace some of the first disparity values with values more representative of structural features present in the image data descriptive of the scene [which have been degraded due to noise or low spatial resolution,]
- the guided filtering operation performed by applying a series of weighted median filter operations to pixel intensity values in the sequence of first disparity values so that each median filter operation replaces a member in the sequence of first disparity values with a median intensity value, where each median intensity value is based on intensity values in a group of pixels within a window of pixels positioned about said member in the sequence,
- each window being of a variable size to include a variable number of pixels positioned about said member in the sequence, where selections of multiple ones of the window sizes are based on a measure of similarity between the first disparity values and said other image data, and wherein the series of [weighted] median filter operations provides a new sequence of disparity values for a refined disparity map or from which a depth map of improved accuracy can be created.
A method for performing a [median] filter operation on a sequence of data [intensity or magnitude] values which are a subset in a larger plurality of data values derived from acquisitions with one or more devices, the method comprising:
- performing a series of filter operations on at least some values in the sequence and replacing each in a plurality of the values in the sequence with a new value resulting from the filter operation,
- where each new value is based on values in a different group containing a number of other data values in the sequence and where the number of data values in each group of values is variable and definable in terms of a selectable dimension in a range of window dimensions, and where the window dimensions vary among the groups based on a measure of similarity between values in the sequence and other values in the larger plurality of data values.
A method of defining a series of sizes of adaptive windows, each for use in a filter operation performed on data values for a disparity map, D, corresponding to a scene represented by a reference image I, each filter operation replacing a different data value associated with the disparity map with a new value to improve depth map accuracy for the scene, comprising:
- calculating data values for the disparity map associated with the scene;
- providing data values for the reference image;
- providing a plurality of values, S, collectively corresponding to a similarity map, each value, S, indicative of a level of similarity between a portion of the disparity map data values and a corresponding portion of the reference image data values;
- applying the values, S, to determine a series of windows of variable sizes, each window in the series for use in a different filter operation performed on the disparity map data values;
- with W designating current window size under evaluation:
- defining a smallest window size Wmin of dimension (2j+1) by (2j+1) where j=jmin;
- defining a largest window size Wmax of dimension (2j+1) by (2j+1) where j=jmax;
- for each pixel position Pi,j in the disparity map, performing a test with the inequality statement
- SSIMmin<SSIMmed<SSIMmax
- for each corresponding pixel position Si,j in the similarity map beginning with the smallest window size, where:
- SSIMmin is the minimum pixel value inside the current window, W
- SSIMmax is the maximum pixel value inside the current window, W and
- SSIMmed is the median pixel value determined for the current window, W;
- and
- concluding that:
- If the inequality statement is true, then the current window size is the chosen window size for performing a weighted median operation about the pixel position Pi,j in the disparity map corresponding to the similarity map pixel position Si,j;
- and
- If the inequality statement in step a) is not true, increasing the size of the window by incrementing j to j+1;
- and
- repeating steps a) and b) until either:
- (i) SSIMmed is between SSIMmin and SSIMmax, in which case the current window size is the chosen window size for performing a weighted median operation about the pixel position Pi,j in the disparity map corresponding to the similarity map pixel position Si,j, or
- (ii) the maximum window size is reached, in which case the maximum window size is the chosen window size for performing the weighted median operation about the pixel position Pi,j in the disparity map corresponding to the similarity map pixel position Si,j.
A method for improving accuracy of depth map information derived from image data descriptive of a scene, comprising:
- performing a matching cost optimization by iteratively refining disparities between corresponding pixels in the image data and using optimization results to create a sequence of initial disparity map data values for an initial disparity map for the scene;
- creating a Mutual Feature Map data based on initial disparity map data values and the image data descriptive of the scene;
- applying the Mutual Feature Map data to create a series of weighting functions representing structural details that can be transferred to the first disparity values to restore degraded features or replace some of the first disparity values with values more representative of structural features present in the image data descriptive of the scene, including applying a series of weighted median filter operations in which a weight kernel based on the Mutual Feature Map data is applied to assign a weighting factor to each pixel position in a weighted median filter operation, whereby each median filter operation replaces a member in the sequence of first disparity values with a median intensity value, where each median intensity value is based on intensity values in a group of pixels within a window of pixels positioned about a data value in the sequence.
Dependencies are inferred from the claim text.