# US 2013/0155050 A1 · Refinement of Depth Maps by Fusion of Multiple Estimates > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0155050 A1 - Document ID: US20130155050A1 - Status: Published application - Kind: A1 - Application number: 13/332,139 - Filed: 2011-12-20 - Published: 2013-06-20 - Earliest claimed: 2011-12-20 - Assignees: Adobe Systems Incorporated - Inventors: Anubha Rastogi; Balaji Krishnamurthy - Web page: https://patentgrep.com/patent/US20130155050A1 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 15/40: Hidden part removal - H04N 25/671: for non-uniformity detection or correction - G06T 2207/10012: Stereo images - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method, system, and computer-readable storage medium are disclosed for generating fused depth maps. A plurality of initial depth maps are generated for a first image of a stereo pair, using at least two separate and distinct techniques for depth estimation. A plurality of initial depth maps are generated for a second image of the stereo pair, using at least two separate and distinct techniques for depth estimation. A fused depth map is generated for the first image based on the plurality of initial depth maps for the first image and the plurality of initial depth maps for the second image. A fused depth map is generated for the second image based on the plurality of initial depth maps for the second image and the plurality of initial depth maps for the first image. ## Claims ### Claim 1 A computer-implemented method, comprising: generating a plurality of initial depth maps for a first image of a stereo pair, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation; generating a plurality of initial depth maps for a second image of the stereo pair, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation; generating a fused depth map for the first image based on the plurality of initial depth maps for the first image and the plurality of initial depth maps for the second image; and generating a fused depth map for the second image based on the plurality of initial depth maps for the second image and the plurality of initial depth maps for the first image. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein generating the fused depth map for the first image comprises: generating a plurality of reliability values for the fused depth map for the first image, wherein each of the plurality of reliability values for the fused depth map for the first image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the first image and in each of the plurality of initial depth maps for the second image, wherein a depth value for each corresponding pixel in the fused depth map for the first image is determined by the reliability values for the fused depth map for the first image; and wherein generating the fused depth map for the second image comprises: generating a plurality of reliability values for the fused depth map for the second image, wherein each of the plurality of reliability values for the fused depth map for the second image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the second image and in each of the plurality of initial depth maps for the first image, wherein a depth value for each corresponding pixel in the fused depth map for the second image is determined by the reliability values for the fused depth map for the second image. ### Claim 3 (depends on 2) The method as recited in claim 2, wherein, in generating the plurality of reliability values for the fused depth map for the first image and in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value is incremented by a smoothness value of the respective depth value. ### Claim 4 (depends on 2) The method as recited in claim 2, wherein, in generating the plurality of reliability values for the fused depth map for the first image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the first image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the second image; and wherein, in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the second image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the first image. ### Claim 5 (depends on 1) The method as recited in claim 1, further comprising: assigning a weight to each depth value in each of the initial depth maps for the first image; and assigning a weight to each depth value in each of the initial depth maps for the second image. ### Claim 6 (depends on 5) The method as recited in claim 5, wherein the weight for each depth value is determined based on a smoothness value and a confirmation value. ### Claim 7 (depends on 6) The method as recited in claim 6, wherein generating the fused depth map for the first image comprises: for each pixel in the fused depth map for the first image, selecting the depth value having the highest weight from the initial depth maps for the first image; and wherein generating the fused depth map for the second image comprises: for each pixel in the fused depth map for the second image, selecting the depth value having the highest weight from the initial depth maps for the second image. ### Claim 8 A system, comprising: at least one processor; and a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to: generate a plurality of initial depth maps for a first image of a stereo pair, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation; generate a plurality of initial depth maps for a second image of the stereo pair, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation; generate a fused depth map for the first image based on the plurality of initial depth maps for the first image and the plurality of initial depth maps for the second image; and generate a fused depth map for the second image based on the plurality of initial depth maps for the second image and the plurality of initial depth maps for the first image. ### Claim 9 (depends on 8) The system as recited in claim 8, wherein, in generating the fused depth map for the first image, the program instructions are executable by the at least one processor to: generate a plurality of reliability values for the fused depth map for the first image, wherein each of the plurality of reliability values for the fused depth map for the first image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the first image and in each of the plurality of initial depth maps for the second image, wherein a depth value for each corresponding pixel in the fused depth map for the first image is determined by the reliability values for the fused depth map for the first image; and wherein, in generating the fused depth map for the second image, the program instructions are executable by the at least one processor to: generate a plurality of reliability values for the fused depth map for the second image, wherein each of the plurality of reliability values for the fused depth map for the second image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the second image and in each of the plurality of initial depth maps for the first image, wherein a depth value for each corresponding pixel in the fused depth map for the second image is determined by the reliability values for the fused depth map for the second image. ### Claim 10 (depends on 9) The system as recited in claim 9, wherein, in generating the plurality of reliability values for the fused depth map for the first image and in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value is incremented by a smoothness value of the respective depth value. ### Claim 11 (depends on 9) The system as recited in claim 9, wherein, in generating the plurality of reliability values for the fused depth map for the first image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the first image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the second image; and wherein, in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the second image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the first image. ### Claim 12 (depends on 8) The system as recited in claim 8, wherein the program instructions are executable by the at least one processor to: assign a weight to each depth value in each of the initial depth maps for the first image; and assign a weight to each depth value in each of the initial depth maps for the second image;. ### Claim 13 (depends on 12) The system as recited in claim 12, wherein the weight for each depth value is determined based on a smoothness value and a confirmation value. ### Claim 14 (depends on 13) The system as recited in claim 13, wherein, in generating the fused depth map for the first image, the program instructions are executable by the at least one processor to: select the depth value having the highest weight from the initial depth maps for the first image for each pixel in the fused depth map for the first image; and wherein, in generating the fused depth map for the second image, the program instructions are executable by the at least one processor to: select the depth value having the highest weight from the initial depth maps for the second image for each pixel in the fused depth map for the second image. ### Claim 15 A computer-readable storage medium storing program instructions computer-executable to perform: generating a plurality of initial depth maps for a first image of a stereo pair, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation; generating a plurality of initial depth maps for a second image of the stereo pair, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation; generating a fused depth map for the first image based on the plurality of initial depth maps for the first image and the plurality of initial depth maps for the second image; and generating a fused depth map for the second image based on the plurality of initial depth maps for the second image and the plurality of initial depth maps for the first image. ### Claim 16 (depends on 15) The computer-readable storage medium as recited in claim 15, wherein generating the fused depth map for the first image comprises: generating a plurality of reliability values for the fused depth map for the first image, wherein each of the plurality of reliability values for the fused depth map for the first image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the first image and in each of the plurality of initial depth maps for the second image, wherein a depth value for each corresponding pixel in the fused depth map for the first image is determined by the reliability values for the fused depth map for the first image; and wherein generating the fused depth map for the second image comprises: generating a plurality of reliability values for the fused depth map for the second image, wherein each of the plurality of reliability values for the fused depth map for the second image is generated on behalf of each respective pixel in each of the plurality of initial depth maps for the second image and in each of the plurality of initial depth maps for the first image, wherein a depth value for each corresponding pixel in the fused depth map for the second image is determined by the reliability values for the fused depth map for the second image. ### Claim 17 (depends on 16) The computer-readable storage medium as recited in claim 16, wherein, in generating the plurality of reliability values for the fused depth map for the first image and in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value is incremented by a smoothness value of the respective depth value. ### Claim 18 (depends on 16) The computer-readable storage medium as recited in claim 16, wherein, in generating the plurality of reliability values for the fused depth map for the first image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the first image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the second image; and wherein, in generating the plurality of reliability values for the fused depth map for the second image, each of the plurality of reliability values for a respective depth value in a respective initial depth map for the second image is incremented by a confirmation weight if the respective depth value is similar to a corresponding depth value in one of the initial depth maps for the first image. ### Claim 19 (depends on 15) The computer-readable storage medium as recited in claim 15, assigning a weight to each depth value in each of the initial depth maps for the first image; and assigning a weight to each depth value in each of the initial depth maps for the second image; wherein the weight for each depth value is determined based on a smoothness value and a confirmation value. ### Claim 20 (depends on 19) The computer-readable storage medium as recited in claim 19, wherein generating the fused depth map for the first image comprises: for each pixel in the fused depth map for the first image, selecting the depth value having the highest weight from the initial depth maps for the first image; and wherein generating the fused depth map for the second image comprises: for each pixel in the fused depth map for the second image, selecting the depth value having the highest weight from the initial depth maps for the second image.