# US 9,438,816 B2 · Adaptive image acquisition for multiframe reconstruction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,438,816 B2 - Document ID: US9438816B2 - Status: Granted - Kind: B2 - Application number: 14/551,998 - Filed: 2014-11-24 - Published: 2015-09-17 - Granted: 2016-09-06 - Earliest claimed: 2008-03-26 - Assignees: Ricoh Company, Ltd. - Applicants: Ricoh Co., Ltd. - Inventors: M. Dirk Robinson; Junlan Yang; David G. Stork - Web page: https://patentgrep.com/patent/US9438816B2 ## CPC - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - G06T 1/0007: {Image acquisition} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/571: from focus - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - H04N 23/673: {based on contrast or high frequency components of image signals, e.g. hill climbing method} - H04N 23/70: Circuitry for compensating brightness variation in the scene - H04N 23/72: Combination of two or more compensation controls - H04N 23/743: Bracketing, i.e. taking a series of images with varying exposure conditions - G06T 2207/10004: Still image; Photographic image - G06T 2207/20208: High dynamic range [HDR] image processing - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Multiframe reconstruction combines a set of acquired images into a reconstructed image. Here, which images to acquire are selected based at least in part on the content of previously acquired images. In one approach, a set of at least three images of an object are acquired at different acquisition settings. For at least one of the images in the set, the acquisition setting for the image is determined based at least in part on the content of previously acquired images. Multiframe image reconstruction is applied to the set of acquired images to synthesize a reconstructed image of the object. ## Claims ### Claim 1 A method for multiframe reconstruction of an object comprising: acquiring a set of at least two images of the object, the images acquired at different acquisition settings; for at least one of the images, determining the acquisition setting for the image based at least in part on improving a depth estimation calculated from the set of images, wherein determining the acquisition setting also takes into account content of previously acquired images as such content relates to depth estimation; and applying a multiframe reconstruction to the set of images to synthesize a single image, the synthesis including calculating a depth estimation for the object. ### Claim 2 (depends on 1) The method of claim 1 wherein the steps of acquiring a set of images and determining an acquisition setting comprise: acquiring at least two initial images at acquisition settings that do not depend on content of previously acquired images; and for every image acquired after the initial images, determining the acquisition setting for the image based at least in part on improving a depth estimation calculated from the set of images, wherein determining the acquisition setting also takes into account content of previously acquired images as such content relates to depth estimation. ### Claim 3 (depends on 2) The method of claim 2 wherein, for every image acquired after the initial images, the step of determining the acquisition setting for said image does not depend on whether any additional images will be acquired after said image. ### Claim 4 (depends on 2) The method of claim 2 wherein, for at least one of the images acquired after the initial images, the step of determining the acquisition setting for said image is based at least in part on assuming that at least one additional image will be acquired after said image. ### Claim 5 (depends on 2) The method of claim 2 wherein the set of images contains K images; and, for every image acquired after the initial images, the step of determining the acquisition setting for said image is based at least in part on assuming a final set containing K images. ### Claim 6 (depends on 1) The method of claim 1 wherein the step of determining the acquisition setting for said image comprises determining the acquisition setting based at least in part on increasing new depth information captured by said image, compared to depth information already captured by previously acquired images. ### Claim 7 (depends on 1) The method of claim 1 wherein the step of determining the acquisition setting for the image further comprises: determining the acquisition setting for said image based at least in part on reducing a total power consumption. ### Claim 8 (depends on 1) The method of claim 1 wherein the step of determining the acquisition setting for said image is based on modeling a relationship between the acquisition settings and an optical point spread function. ### Claim 9 (depends on 1) The method of claim 1 wherein the acquisition setting includes an aperture setting. ### Claim 10 (depends on 1) The method of claim 1 wherein the acquisition setting includes a focus setting. ### Claim 11 (depends on 1) The method of claim 1 wherein the acquisition setting includes an exposure setting. ### Claim 12 (depends on 1) The method of claim 1 wherein the object is modeled as a multi-depth object. ### Claim 13 (depends on 12) The method of claim 12 wherein the step of determining the acquisition setting comprises: estimating different object depths for different tiles of the multi-depth object, based at least in part on content of previously acquired images; and determining the acquisition setting for the image based at least in part on the estimated object depths. ### Claim 14 (depends on 13) The method of claim 13 wherein the step of determining the acquisition setting for the image is based at least in part on a weighted average over the tiles of a merit function evaluated for each tile. ### Claim 15 (depends on 1) The method of claim 1 wherein the selected acquisition setting is selected based on a Maximum-Likelihood principle. ### Claim 16 (depends on 1) The method of claim 1 wherein the selected acquisition setting is characterized by a modulation transfer function (MTF), where each zero crossing of the MTF does not occur at a spatial frequency at which MTFs for all the prior acquisition settings also have zero crossings. ### Claim 17 (depends on 1) The method of claim 1 wherein the acquisition setting is selected based also on factors not relating to a quality of the synthesized single image or an accuracy of the calculated depth estimation. ### Claim 18 A non-transitory computer readable medium containing software instructions that, when executed on a computerized system, cause the computerized system to: cause a camera to acquire a set of at least two images of the object, the images acquired at different acquisition settings; for at least one of the images, determine the acquisition setting for the image based at least in part on improving a depth estimation calculated from the set of images, wherein determining the acquisition setting also takes into account content of previously acquired images as such content relates to depth estimation; and apply a multiframe reconstruction to the set of images to synthesize a single image, the synthesis including calculating a depth estimation for the object. ### Claim 19 A system comprising: a camera that acquires images of an object; a camera controller that controls the camera to acquire a set of at least two images of the object at different acquisition settings; and an adaptive acquisition module communicating with the camera controller wherein, for at least one of the images, the adaptive acquisition module determines the acquisition setting for the image based at least in part on improving a depth estimation calculated from the set of images, wherein determining the acquisition setting also takes into account content of previously acquired images as such content relates to depth estimation. ### Claim 20 (depends on 19) The system of claim 19 further comprising: a multiframe reconstruction module that applies a multiframe reconstruction to the set of images to synthesize a single image, the synthesis including calculating a depth estimation for the object.