# US 2014/0152647 A1 · Depth Estimation Using Three-Dimensional Epipolar Data Structures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0152647 A1 - Document ID: US20140152647A1 - Status: Published application - Kind: A1 - Application number: 13/690,514 - Filed: 2012-11-30 - Published: 2014-06-05 - Earliest claimed: 2012-11-30 - Assignees: Adobe Systems Incorporated - Applicants: Adobe Systems Incorporated - Inventors: Michael Wish Tao; Sunil Hadap - Web page: https://patentgrep.com/patent/US20140152647A1 ## CPC - G06T 7/557: from light fields, e.g. from plenoptic cameras - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are provided for depth map estimation using three-dimensional epipolar data structures. The image manipulation application receives image data depicting an image space from a multiple perspectives. The image manipulation application generates at least one three-dimensional epipolar data structure from the image data. The at least one three-dimensional epipolar data structure includes data describing the difference in position of at least one object between the perspectives. The at least one three-dimensional epipolar data structure corresponds to at least one region of the image space. The image manipulation application generates a depth map based on the at least one three-dimensional epipolar data structure. ## Claims ### Claim 1 A method comprising: receiving, by an image manipulation application executed by a processing device, image data depicting an image space from a plurality of perspectives; generating, by the image manipulation application, at least one three-dimensional epipolar data structure from the image data, the at least one three-dimensional epipolar data structure comprising data describing a difference in position of at least one object between the plurality of perspectives, wherein the at least one three-dimensional epipolar data structure corresponds to at least one region of the image space; and generating, by the image manipulation application, a depth map based on the at least one three-dimensional epipolar data structure. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the image data depicting the image space from the plurality of perspectives comprises: receiving an input image comprising the image data, wherein the input image comprises a plurality of super-pixels, each super-pixel comprising image data encoded from multiple perspectives; and extracting the image data depicting the image space from the plurality of perspectives from at least some of the plurality of super-pixels. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the at least one three-dimensional epipolar data structure comprises: identifying the at least one region of the image space; generating a plurality of two-dimensional epipolar data structures, wherein each of the plurality of two-dimensional epipolar data structures corresponds to at least a portion of the at least one region of the image space, wherein at least one first two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a first difference in position of the at least one object along a first axis and at least one second two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a second difference in position of the at least one object along a second axis; and generating the at least one three-dimensional epipolar data structure based on the plurality of two-dimensional epipolar data structures. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the depth map based on the at least one three-dimensional epipolar data structure comprises: determining a function describing a relationship between at least two instances of the at least one object in the at least one three-dimensional epipolar data structure; determining a derivative of the function; and estimating a depth value for the at least one object based on the derivative of the function. ### Claim 5 (depends on 1) The method of claim 1, further comprising: generating a confidence map associated with the depth map based on the at least one three-dimensional epipolar data structure comprises, wherein the confidence map identifies a plurality of high-confidence regions of the image space associated with a plurality depth values of the depth map generated based on the at least one three-dimensional epipolar data structure and a plurality of low-confidence regions of the image space; and estimating an additional plurality of depth values corresponding to the plurality of low-confidence regions, wherein each of the additional plurality of depth values is estimated based on at least one of the plurality depth values associated with a corresponding one of the plurality of high-confidence regions. ### Claim 6 (depends on 5) The method of claim 5, wherein estimating the additional plurality of depth values comprises, for each of the plurality of low-confidence regions: identifying at least two of the plurality of high-confidence regions adjacent to the respective low confidence region from the confidence map; averaging at least two depth values respectively corresponding to the at least two of the plurality of high-confidence regions; and selecting the average depth value as a respective depth value for the respective low confidence region. ### Claim 7 (depends on 1) The method of claim 1, further comprising: capturing, by an imaging device in which the processing device is disposed, the image data; and storing, by the imaging device, the image data to a non-transitory computer-readable medium accessible by the processing device. ### Claim 8 A non-transitory computer-readable medium embodying program code executable by a processing device, the non-transitory computer-readable medium comprising: program code for receiving image data depicting an image space from a plurality of perspectives; program code for generating at least one three-dimensional epipolar data structure from the image data, the at least one three-dimensional epipolar data structure comprising data describing a difference in position of at least one object between the plurality of perspectives, wherein the at least one three-dimensional epipolar data structure corresponds to at least one region of the image space; and program code for generating a depth map based on the at least one three-dimensional epipolar data structure. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein receiving the image data depicting the image space from the plurality of perspectives comprises: receiving an input image comprising the image data, wherein the input image comprises a plurality of super-pixels, each super-pixel comprising image data encoded from multiple perspectives; and extracting the image data depicting the image space from the plurality of perspectives from at least some of the plurality of super-pixels. ### Claim 10 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the program code generating the at least one three-dimensional epipolar data structure comprises: program code for identifying the at least one region of the image space; program code for generating a plurality of two-dimensional epipolar data structures, wherein each of the plurality of two-dimensional epipolar data structures corresponds to at least a portion of the at least one region of the image space, wherein at least one first two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a first difference in position of the at least one object along a first axis and at least one second two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a second difference in position of the at least one object along a second axis; and program code for generating the at least one three-dimensional epipolar data structure based on the plurality of two-dimensional epipolar data structures. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the program code for generating the depth map based on the at least one three-dimensional epipolar data structure comprises: program code for determining a function describing a relationship between at least two instances of the at least one object in the at least one three-dimensional epipolar data structure; program code for determining a derivative of the function; and program code for estimating a depth value for the at least one object based on the derivative of the function. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, further comprising: program code for generating a confidence map associated with the depth map based on the at least one three-dimensional epipolar data structure comprises, wherein the confidence map identifies a plurality of high-confidence regions of the image space associated with a plurality depth values of the depth map generated based on the at least one three-dimensional epipolar data structure and a plurality of low-confidence regions of the image space; and program code for estimating an additional plurality of depth values corresponding to the plurality of low-confidence regions, wherein each of the additional plurality of depth values is estimated based on at least one of the plurality depth values associated with a corresponding one of the plurality of high-confidence regions. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the program code for estimating the additional plurality of depth values comprises, for each of the plurality of low-confidence regions: program code for identifying at least two of the plurality of high-confidence regions adjacent to the respective low confidence region from the confidence map; program code for averaging at least two depth values respectively corresponding to the at least two of the plurality of high-confidence regions; and program code for selecting the average depth value as a respective depth value for the respective low confidence region. ### Claim 14 A system comprising: a processor configured to execute instructions stored in a non-transitory computer-readable medium; wherein the instructions comprise an image manipulation application configured to perform operations comprising: receiving image data depicting an image space from a plurality of perspectives; generating at least one three-dimensional epipolar data structure from the image data, the at least one three-dimensional epipolar data structure comprising data describing a difference in position of at least one object between the plurality of perspectives, wherein the at least one three-dimensional epipolar data structure corresponds to at least one region of the image space; and generating a depth map based on the at least one three-dimensional epipolar data structure. ### Claim 15 (depends on 14) The system of claim 14, further comprising an imaging device configured to record the image data, wherein the processor is configured to receive the image data from the imaging device. ### Claim 16 (depends on 14) The system of claim 14, wherein receiving the image data depicting the image space from the plurality of perspectives comprises: receiving an input image comprising the image data, wherein the input image comprises a plurality of super-pixels, each super-pixel comprising image data encoded from multiple perspectives; and extracting the image data depicting the image space from the plurality of perspectives from at least some of the plurality of super-pixels. ### Claim 17 (depends on 14) The system of claim 14, wherein generating the at least one three-dimensional epipolar data structure comprises: identifying the at least one region of the image space; generating a plurality of two-dimensional epipolar data structures, wherein each of the plurality of two-dimensional epipolar data structures corresponds to at least a portion of the at least one region of the image space, wherein at least one first two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a first difference in position of the at least one object along a first axis and at least one second two-dimensional epipolar data structure of the plurality of two-dimensional epipolar data structures comprises data describing a second difference in position of the at least one object along a second axis; and generating the at least one three-dimensional epipolar data structure based on the plurality of two-dimensional epipolar data structures. ### Claim 18 (depends on 14) The system of claim 14, wherein generating the depth map based on the at least one three-dimensional epipolar data structure comprises: determining a function describing a relationship between at least two instances of the at least one object in the at least one three-dimensional epipolar data structure; determining a derivative of the function; and estimating a depth value for the at least one object based on the derivative of the function. ### Claim 19 (depends on 14) The system of claim 14, wherein the image manipulation application is further configured to perform additional operations comprising: generating a confidence map associated with the depth map based on the at least one three-dimensional epipolar data structure comprises, wherein the confidence map identifies a plurality of high-confidence regions of the image space associated with a plurality depth values of the depth map generated based on the at least one three-dimensional epipolar data structure and a plurality of low-confidence regions of the image space; and estimating an additional plurality of depth values corresponding to the plurality of low-confidence regions, wherein each of the additional plurality of depth values is estimated based on at least one of the plurality depth values associated with a corresponding one of the plurality of high-confidence regions. ### Claim 20 (depends on 19) The system of claim 19, wherein estimating the additional plurality of depth values comprises, for each of the plurality of low-confidence regions: identifying at least two of the plurality of high-confidence regions adjacent to the respective low confidence region from the confidence map; averaging at least two depth values respectively corresponding to the at least two of the plurality of high-confidence regions; and selecting the average depth value as a respective depth value for the respective low confidence region.