# US 11,113,832 B2 · Aperture supervision for single-view depth prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,832 B2 - Document ID: US11113832B2 - Status: Granted - Kind: B2 - Application number: 16/759,808 - Filed: 2017-11-03 - Published: 2021-06-17 - Granted: 2021-09-07 - Earliest claimed: 2017-11-03 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Neal Wadhwa; Jonathan Barron; Rahul Garg; Pratul Srinivasan - Web page: https://patentgrep.com/patent/US11113832B2 ## CPC - G01S 11/12: using electromagnetic waves other than radio waves - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/70: Denoising; Smoothing - G06T 7/557: from light fields, e.g. from plenoptic cameras - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10052: Images from lightfield camera - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Example embodiments allow for training of artificial neural networks (ANNs) to generate depth maps based on images. The ANNs are trained based on a plurality of sets of images, where each set of images represents a single scene and the images in such a set of images differ with respect to image aperture and/or focal distance. An untrained ANN generates a depth map based on one or more images in a set of images. This depth map is used to generate, using the image(s) in the set, a predicted image that corresponds, with respect to image aperture and/or focal distance, to one of the images in the set. Differences between the predicted image and the corresponding image are used to update the ANN. ANNs tramed in this manner are especially suited for generating depth maps used to perform simulated image blur on small-aperture images. ## Claims ### Claim 1 A method comprising: obtaining a plurality of images, wherein a set of at least two images of the plurality of images describe a common scene, wherein the set of at least two images comprises a source image having a first depth-of-field and a target image having a second depth-of-field, wherein the second depth-of-field is less than the first depth-of-field; determining, using an artificial neural network, a depth map for the common scene based on the source image; determining, based on the determined depth map for the common scene, a predicted image based on the source image such that the predicted image has a depth-of-field corresponding to the second depth-of-field; determining a difference between the predicted image and the target image; updating the artificial neural network based on the determined difference; obtaining an image of a scene of interest; and using the updated artificial neural network to generate a depth map for the scene of interest based on the image of the scene of interest. ### Claim 2 (depends on 1) The method of claim 1, wherein determining, based on the first depth map, a predicted image based on the source image comprises using a differentiable aperture rendering function. ### Claim 3 (depends on 2) The method of claim 2, wherein the using the differentiable aperture rendering function to determine the predicted image comprises: determining an estimated light field based on the source image; and based on the first depth map, shearing and projecting the estimated light field to determine the predicted image. ### Claim 4 (depends on 1) The method of claim 1, wherein determining, using the artificial neural network, the depth map for the common scene based on the source image comprises: determining, using the artificial neural network, a set of depth values based on the source image, wherein each depth value of the set of depth values corresponds to a respective location within the source image; and upsampling the set of depth values to generate the depth map for the common scene. ### Claim 5 (depends on 4) The method of claim 4, wherein upsampling the set of depth values to generate the depth map for the common scene comprises using a bilateral method to upsample the set of depth values based on the source image. ### Claim 6 (depends on 1) The method of claim 1, wherein the target image is a first target image, wherein the predicted image is a first predicted image, wherein the determined difference is a first determined difference, wherein the set of at least two images further comprises a second target image having a third depth-of-field, wherein the third depth-of-field differs from the second depth-of-field and is less than the first depth-of-field, the method further comprising: determining, based on the determined depth map for the common scene, a second predicted image based on the source image such that the second predicted image has a depth-of-field corresponding to the third depth-of-field; and determining a second difference between the second predicted image and the second target image, wherein updating the artificial neural network comprises updating the neural network based on the second difference. ### Claim 7 (depends on 1) The method of claim 1, wherein obtaining the set of at least two images that describe the common scene comprises: capturing, using a light field camera, a light field from the common scene; generating the source image based on the captured light field such that the source image has the first depth-of-field; and generating the target image based on the captured light field such that the target image has the second depth-of-field. ### Claim 8 (depends on 1) The method of claim 1, wherein obtaining the set of at least two images that describe the common scene comprises: capturing, using a camera set to a first aperture setting, the source image; and capturing, using the camera set to a second aperture setting, the target image, wherein the second aperture setting is wider than the first aperture setting. ### Claim 9 (depends on 1) The method of claim 1, wherein obtaining the image of the scene of interest comprises operating a cell phone to capture the image of the scene of interest, the method further comprising: transmitting, from a server to the cell phone, an indication of the updated artificial neural network, wherein using the updated artificial neural network to generate the depth map for the scene of interest based on the image of the scene of interest comprises a processor of the cell phone using the updated artificial neural network to generate the depth map for the scene of interest. ### Claim 10 (depends on 1) The method of claim 1, further comprising: performing image processing on the image of the scene of interest based on the determined depth map for the scene of interest. ### Claim 11 (depends on 1) The method of claim 1, wherein the artificial neural network is a convolutional neural network. ### Claim 12 A method comprising: obtaining, by a system, a plurality of images, wherein a set of at least two images of the plurality of images describe a common scene, wherein the set of at least two images comprises a source image having a first depth-of-field and a target image having a second depth-of-field, wherein the second depth-of-field is less than the first depth-of-field; determining, by the system using an artificial neural network, a depth map for the common scene based on the source image; determining, by the system based on the determined depth map for the common scene, a predicted image based on the source image such that the predicted image has a depth-of-field corresponding to the second depth-of-field; determining, by the system, a difference between the predicted image and the target image; updating, by the system, the artificial neural network based on the determined difference; and transmitting, from the system to a remote device, an indication of the updated artificial neural network. ### Claim 13 (depends on 12) The method of claim 12, wherein determining, based on the first depth map, a predicted image based on the source image comprises using a differentiable aperture rendering function. ### Claim 14 (depends on 13) The method of claim 13, wherein the using the differentiable aperture rendering function to determine the predicted image comprises: determining an estimated light field based on the source image; and based on the first depth map, shearing and projecting the estimated light field to determine the predicted image. ### Claim 15 (depends on 12) The method of claim 12, wherein the target image is a first target image, wherein the predicted image is a first predicted image, wherein the determined difference is a first determined difference, wherein the set of at least two images further comprises a second target image having a third depth-of-field, wherein the third depth-of-field differs from the second depth-of-field and is less than the first depth-of-field, the method further comprising: determining, based on the determined depth map for the common scene, a second predicted image based on the source image such that the second predicted image has a depth-of-field corresponding to the third depth-of-field; and determining a second difference between the second predicted image and the second target image, wherein updating the artificial neural network comprises updating the neural network based on the second difference. ### Claim 16 (depends on 12) The method of claim 12, wherein the artificial neural network is a convolutional neural network. ### Claim 17 A method comprising: obtaining a plurality of images of a scene, wherein the images each have a shallow depth-of-field and differ with respect to focal distance; determining, using an artificial neural network, a depth map for the scene based on the plurality of images; determining, based on the plurality of images and the determined depth map for the scene, a predicted large depth-of-field image of the scene; determining, based on the determined depth map for the common scene, a predicted shallow depth-of-field image based on the predicted large depth-of-field image such that the predicted shallow depth-of-field image has a focal distance corresponding to a particular image of the plurality of images; determining a difference between the predicted shallow depth-of-field image and the particular image; updating the artificial neural network based on the determined difference; and using the updated artificial neural network to determine an updated depth map for the scene based on the plurality of images. ### Claim 18 (depends on 17) The method of claim 17, wherein the artificial neural network is an autoencoder convolutional neural network. ### Claim 19 (depends on 17) The method of claim 17, wherein determining, based on the determined depth map for the common scene, a predicted shallow depth-of-field image based on the predicted large depth-of-field image such that the predicted shallow depth-of-field image has a focal distance corresponding to the particular image of the plurality of images comprises using a differentiable aperture rendering function. ### Claim 20 (depends on 17) The method of claim 17, wherein obtaining the plurality of images of the scene comprises operating a camera to capture, using respective different focal distance settings, each image of the plurality of images of the scene.