# US 9,521,391 B2 · Settings of a digital camera for depth map refinement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,521,391 B2
- Document ID: US9521391B2
- Status: Granted
- Kind: B2
- Application number: 15/056,283
- Filed: 2016-02-29
- Published: 2016-06-23
- Granted: 2016-12-13
- Earliest claimed: 2014-12-19
- Assignees: Adobe Systems Incorporated
- Applicants: Adobe Systems Incorporated
- Inventors: Huixuan Tang; Scott Cohen; Stephen Schiller; Brian Price
- Web page: https://patentgrep.com/patent/US9521391B2

## CPC

- G06T 5/73: Deblurring; Sharpening
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/571: from focus
- H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)}
- H04N 13/128: Adjusting depth or disparity
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 17/002: {for television cameras}
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- H04N 23/676: {Bracketing for image capture at varying focusing conditions}
- H04N 23/689: {Motion occurring during a rolling shutter mode}
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10148: Varying focus
- G06T 2207/20048: Transform domain processing
- H04N 1/387: Composing, repositioning or otherwise {geometrically} modifying originals
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N 2213/003: Aspects relating to the "2D+depth" image format
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods are disclosed for identifying depth refinement image capture instructions for capturing images that may be used to refine existing depth maps. The depth refinement image capture instructions are determined by evaluating, at each image patch in an existing image corresponding to the existing depth map, a range of possible depth values over a set of configuration settings. Each range of possible depth values corresponds to an existing depth estimate of the existing depth map. This evaluation enables selection of one or more configuration settings in a manner such that there will be additional depth information derivable from one or more additional images captured with the selected configuration settings. When a refined depth map is generated using the one or more additional images, this additional depth information is used to increase the depth precision for at least one depth estimate from the existing depth map.

## Claims

### Claim 1

A computer-implemented method comprising:
identifying, by a processing device, image patches of a first image of a scene corresponding to a depth map;
determining, by the processing device and for each image patch, a respective depth range comprising possible depth values;
determining, by the processing device and based on the determined depth ranges, estimated increases in depth precision for respective image patches, the estimated increases in depth perception associated with a modification to configuration settings of a camera;
generating, by the processing device and based on the estimated increases in depth precision, a depth refinement image capture instruction including the modification to configuration settings; and
outputting, by the processing device, the depth refinement image capture instruction usable by the camera when capturing a second image of the scene for refining the depth map.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the depth range is determined based at least in part on at least one of:
a measure of uncertainty representing an accuracy of each depth estimate in the depth map; and
a depth likelihood map indicating a likelihood for of each depth pixel of the depth map.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining the estimated increases in the depth precision for the respective image patches includes:
identifying a blur kernel corresponding to each of the possible depth values and at least one of the configuration settings;
determining, for each blur kernel, a simulated image patch by blurring an all-in-focus image patch of an all-in-focus image with the blur kernel, the simulated image patch corresponding to the image patch of the first image; and
determining, based on the simulated image patches, an amount of possible increase in depth precision for the image patch for the at least one of the configuration settings.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein determining the amount of possible increase in depth precision for the image patch includes:
determining a measure of distance between patch appearances for each pair of simulated image patches corresponding to adjacent possible depth values; and
summing the measures of distance, wherein the sum of the measure of distances corresponds to the amount of possible increase in depth precision for the image patch.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein generating the depth refinement image capture instruction comprises:
estimating, for each pixel in the depth map and for each of a plurality of potential modifications to the configuration settings, an increase in the depth precision the pixel if the configuration setting with the potential modification is used to capture an additional image and additional depth information is derived from the additional image;
generating a voting map, wherein the voting map indicates, for each pixel in the depth map, the degree to which the additional depth information derived from the additional image captured using the potential modifications to the configuration settings is likely to improve the depth estimate at the pixel; and
selecting at least one of the potential modification as the modification for inclusion in the depth refinement image capture instruction.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
capturing the second image in accordance with the depth refinement image capture instruction; and
generating a refined depth map using the second image, wherein the refined depth map includes at least one depth estimate that is more precise than the at least one depth estimate in the depth map.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving an input, the input comprising at least one of a minimum amount of possible increase in depth precision for the depth map, a total quantity of additional images to capture, and a total amount of time for capturing additional images, and wherein generating the depth refinement image capture instruction is performed in accordance with the input.

### Claim 8

A system comprising:
a processing device; and
a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured for executing instructions stored in the non-transitory computer-readable medium and thereby performing operations comprising:
identifying image patches of a first image of a scene corresponding to a depth map;
determining, for each image patch, a respective depth range comprising possible depth values;
determining, based on the determined depth ranges, estimated increases in depth precision for respective image patches, the estimated increases in depth perception associated with a modification to configuration settings of a camera;
generating, based on the estimated increases in depth precision, a depth refinement image capture instruction including the modification to configuration settings; and
outputting the depth refinement image capture instruction usable by the camera when capturing a second image of the scene for refining the depth map.

### Claim 9 (depends on 8)

The system of claim 8, wherein the depth range is determined based at least in part on at least one of:
a measure of uncertainty representing an accuracy of each depth estimate in the depth map; and
a depth likelihood map indicating a likelihood for of each depth pixel of the depth map.

### Claim 10 (depends on 8)

The system of claim 8, wherein determining the estimated increases in the depth precision for the respective image patches includes:
identifying a blur kernel corresponding to each of the possible depth values and at least one of the configuration settings;
determining, for each blur kernel, a simulated image patch by blurring an all-in-focus image patch of an all-in-focus image with the blur kernel, the simulated image patch corresponding to the image patch of the first image; and
determining, based on the simulated image patches, an amount of possible increase in depth precision for the image patch for the at least one of the configuration settings.

### Claim 11 (depends on 10)

The system of claim 10, wherein determining the amount of possible increase in depth precision for the image patch includes:
determining a measure of distance between patch appearances for each pair of simulated image patches corresponding to adjacent possible depth values; and
summing the measures of distance, wherein the sum of the measure of distances corresponds to the amount of possible increase in depth precision for the image patch.

### Claim 12 (depends on 8)

The system of claim 8, wherein generating the depth refinement image capture instruction comprises:
estimating, for each pixel in the depth map and for each of a plurality of potential modifications to the configuration settings, an increase in the depth precision the pixel if the configuration setting with the potential modification is used to capture an additional image and additional depth information is derived from the additional image;
generating a voting map, wherein the voting map indicates, for each pixel in the depth map, the degree to which the additional depth information derived from the additional image captured using the potential modifications to the configuration settings is likely to improve the depth estimate at the pixel; and
selecting at least one of the potential modification as the modification for inclusion in the depth refinement image capture instruction.

### Claim 13 (depends on 8)

The system of claim 8, the operations further comprising:
capturing the second image in accordance with the depth refinement image capture instruction; and
generating a refined depth map using the second image, wherein the refined depth map includes at least one depth estimate that is more precise than the at least one depth estimate in the depth map.

### Claim 14 (depends on 8)

The system of claim 8, the operations further comprising receiving an input, the input comprising at least one of a minimum amount of possible increase in depth precision for the depth map, a total quantity of additional images to capture, and a total amount of time for capturing additional images, and wherein generating the depth refinement image capture instruction is performed in accordance with the input.

### Claim 15

A non-transitory computer-readable medium storing instructions that, when executed by a processing device, configure the processing device to perform operations comprising:
identifying image patches of a first image of a scene corresponding to a depth map;
determining, for each image patch, a respective depth range comprising possible depth values;
determining, based on the determined depth ranges, estimated increases in depth precision for respective image patches, the estimated increases in depth perception associated with a modification to configuration settings of a camera;
generating, based on the estimated increases in depth precision, a depth refinement image capture instruction including the modification to configuration settings; and
outputting the depth refinement image capture instruction usable by the camera when capturing a second image of the scene for refining the depth map.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the depth range is determined based at least in part on at least one of:
a measure of uncertainty representing an accuracy of each depth estimate in the depth map; and
a depth likelihood map indicating a likelihood for of each depth pixel of the depth map.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein determining the estimated increases in the depth precision for the respective image patches includes:
identifying a blur kernel corresponding to each of the possible depth values and at least one of the configuration settings;
determining, for each blur kernel, a simulated image patch by blurring an all-in-focus image patch of an all-in-focus image with the blur kernel, the simulated image patch corresponding to the image patch of the first image; and
determining, based on the simulated image patches, an amount of possible increase in depth precision for the image patch for the at least one of the configuration settings.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein determining the amount of possible increase in depth precision for the image patch includes:
determining a measure of distance between patch appearances for each pair of simulated image patches corresponding to adjacent possible depth values; and
summing the measures of distance, wherein the sum of the measure of distances corresponds to the amount of possible increase in depth precision for the image patch.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein generating the depth refinement image capture instruction comprises:
estimating, for each pixel in the depth map and for each of a plurality of potential modifications to the configuration settings, an increase in the depth precision the pixel if the configuration setting with the potential modification is used to capture an additional image and additional depth information is derived from the additional image;
generating a voting map, wherein the voting map indicates, for each pixel in the depth map, the degree to which the additional depth information derived from the additional image captured using the potential modifications to the configuration settings is likely to improve the depth estimate at the pixel; and
selecting at least one of the potential modification as the modification for inclusion in the depth refinement image capture instruction.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, the operations further comprising:
capturing the second image in accordance with the depth refinement image capture instruction; and
generating a refined depth map using the second image, wherein the refined depth map includes at least one depth estimate that is more precise than the at least one depth estimate in the depth map.