# US 9,872,012 B2 · Method and apparatus for image capturing and simultaneous depth extraction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,872,012 B2 - Document ID: US9872012B2 - Status: Granted - Kind: B2 - Application number: 14/791,887 - Filed: 2015-07-06 - Published: 2016-01-07 - Granted: 2018-01-16 - Earliest claimed: 2014-07-04 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Vladimir Petrovich Paramonov; Ivan Andreevich Panchenko; Victor Valentinovich Bucha - Web page: https://patentgrep.com/patent/US9872012B2 ## CPC - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G01B 11/22: for measuring depth - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 7/50: Depth or shape recovery - H04N 13/218: using spatial multiplexing - H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20228: Disparity calculation for image-based rendering - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system for image capturing and depth extraction includes a camera and a data processor. The camera includes: a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis. The data processor is configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information. ## Claims ### Claim 1 A system for image capturing and depth extraction, the system comprising: a lens system; a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information. ### Claim 2 (depends on 1) The system of claim 1, wherein the different spectrum channels form a basis of the spectrum coded aperture. ### Claim 3 (depends on 2) The system of claim 2, wherein the processing basis is different from the sensor basis and the basis of the spectrum coded aperture. ### Claim 4 (depends on 1) The system of claim 1, wherein the spectrum coded aperture has three regions including a transparent region in a central portion, and two regions having spectrum bandwidths respectively corresponding to yellow and cyan. ### Claim 5 (depends on 1) The system of claim 1, wherein the at least two regions of the spectrum coded aperture have spectrum bandwidths respectively corresponding to yellow and cyan. ### Claim 6 (depends on 1) The system of claim 1, wherein the spectrum coded aperture includes three congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta. ### Claim 7 (depends on 1) The system of claim 1, wherein the spectrum coded aperture includes three non-congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta. ### Claim 8 (depends on 1) The system of claim 1, wherein the spectrum coded aperture has a smooth bandwidth change over an aperture region. ### Claim 9 (depends on 1) The system of claim 1, wherein the spectrum coded aperture is fixed to the lens system. ### Claim 10 (depends on 1) The system of claim 1, wherein the spectrum coded aperture is attachable to and detachable from the lens system. ### Claim 11 (depends on 1) The system of claim 1, wherein the spectrum coded aperture has a combination of an opaque region and a congruent region, and the congruent region is transparent or transmits ultraviolet light, infrared light, or visible light. ### Claim 12 (depends on 1) The system of claim 1, wherein the spectrum coded aperture has a combination of an opaque region and a non-congruent region, and the non-congruent region is transparent or transmits ultraviolet light, infrared light, or visible light. ### Claim 13 (depends on 1) The system of claim 1, wherein the data processor comprises a preprocessing unit configured to perform the converting the captured image, a disparity estimation unit configured to perform the extracting the disparity, and a conversion unit configured to perform the converting the disparity to the depth information. ### Claim 14 (depends on 13) The system of claim 13, wherein the data processor further comprises an image restoration unit configured to restore the captured image based on the extracted disparity. ### Claim 15 A method of image capturing and depth extraction, the method comprising: recording at least two shifted spectrum channels of a light field to form an image captured from a video; converting the captured image into an image of a processing basis; estimating a disparity based on a correlation between pixels of the spectrum channels in the processing basis to extract a disparity map; restoring the captured image based on the extracted disparity map; and converting the disparity map into a depth map. ### Claim 16 (depends on 15) The method of claim 15, wherein the estimating the disparity comprises: generating candidate images having respective shifts in the spectrum channels; computing a matching cost involved in the candidate images in the spectrum channels; propagating a matching cost involved in a low textured region of the candidate images; and estimating a matching cost having a sub-pixel accuracy based on the propagated matching cost. ### Claim 17 (depends on 15) The method of claim 15, wherein the correlation between the pixels of the spectrum channel for requesting the disparity estimation includes a correlation metric computed in a sparse moving window. ### Claim 18 (depends on 15) The method of claim 15, wherein the correlation between the pixels of the spectrum channel for requesting the disparity estimation is computed by using at least one stereo matching algorithm. ### Claim 19 (depends on 15) The method of claim 15, wherein the restoring the captured image comprises performing a spectrum channel alignment in the processing basis. ### Claim 20 A mobile device for image capturing and depth extraction in ultraviolet light, infrared light, or visible light, the mobile device comprising: a lens system; at least one spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and a coded aperture fixture configured to move at least one spectrum coded aperture relatively with respect to the lens system; and a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information. ### Claim 21 (depends on 20) The mobile device of claim 20, wherein the different spectrum channels form a basis of the spectrum coded aperture. ### Claim 22 (depends on 20) The mobile device of claim 20, wherein the processing basis is different from the sensor basis and the basis of the spectrum coded aperture. ### Claim 23 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture has three regions including a transparent region in a central portion, and two regions having spectrum bandwidths respectively corresponding to yellow and cyan. ### Claim 24 (depends on 20) The mobile device of claim 20, wherein the at least two regions of the spectrum coded aperture have spectrum bandwidths respectively corresponding to yellow and cyan. ### Claim 25 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture includes three congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta. ### Claim 26 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture includes three non-congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta. ### Claim 27 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture has a smooth bandwidth change over an aperture region. ### Claim 28 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture is fixed to the lens system. ### Claim 29 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture is attachable to and detachable from the lens system. ### Claim 30 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture has a combination of an opaque region and a congruent region, and the congruent region is transparent or transmits ultraviolet light, infrared light, or visible light. ### Claim 31 (depends on 20) The mobile device of claim 20, wherein the spectrum coded aperture has a combination of an opaque region and a non-congruent region, and the non-congruent region is transparent or transmits ultraviolet light, infrared light, or visible light. ### Claim 32 (depends on 20) The mobile device of claim 20, wherein the data processor comprises a preprocessing unit configured to perform the converting the captured image, a disparity estimation unit configured to perform the extracting the disparity, and a conversion unit configured to perform the converting the disparity to the depth information. ### Claim 33 (depends on 32) The mobile device of claim 32, wherein the data processor further comprises an image restoration unit configured to restore the captured image based on the extracted disparity.