# US 8,786,718 B2 · Image processing apparatus, image capturing apparatus, image processing method and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,786,718 B2 - Document ID: US8786718B2 - Status: Granted - Kind: B2 - Application number: 13/356,078 - Filed: 2012-01-23 - Published: 2012-08-23 - Granted: 2014-07-22 - Earliest claimed: 2011-02-18 - Assignees: Canon Kabushiki Kaisha - Applicants: Keiichi Sawada - Inventors: Keiichi Sawada - Web page: https://patentgrep.com/patent/US8786718B2 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/95: Computational photography systems, e.g. light-field imaging systems - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There is provided an image processing apparatus comprising: an acquisition unit configured to acquire captured images captured by a plurality of image capturing units for capturing an object from different viewpoints; a specifying unit configured to specify a defective image from the plurality of captured images; a determination unit configured to determine a weight for each captured image based on a position of the image capturing unit that has captured the defective image specified by the specifying unit; and a synthesis unit configured to generate a synthesized image by weighting and synthesizing the plurality of captured images based on the weights determined by the determination unit. ## Claims ### Claim 1 An image processing apparatus comprising: an acquisition unit configured to acquire captured images captured by a plurality of image capturing units for capturing an object from different viewpoints; a specifying unit configured to specify a defective image from the plurality of captured images; a determination unit configured to determine a weight for each captured image based on a position of the image capturing unit that has captured the defective image specified by said specifying unit; and a synthesis unit configured to generate a synthesized image by weighted synthesis of the plurality of captured images based on the weights determined by said determination unit, wherein said determination unit is further configured to select at least one captured image different from the defective image as a selected image from the plurality of captured images, and determine the weights for each of the defective image and the selected image to be smaller than a weight of a captured image that is neither the defective image nor the selected image, wherein said determination unit is further configured to determine a relationship between the weights and a reference viewpoint serving as a viewpoint of the synthesized image so that a gravity center of the image capturing units that have not captured the defective image or the selected image is set as the reference viewpoint. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein said specifying unit is further configured to specify the defective image by comparing the plurality of captured images. ### Claim 3 (depends on 1) The apparatus according to claim 1, wherein said determination unit is further configured to set the weight for the defective image to 0. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein said determination unit is further configured to assign the weight to each of the plurality of captured images, and adjust the assigned weight so as to be smaller for the defective image and the selected image. ### Claim 5 (depends on 4) The apparatus according to claim 4, wherein said determination unit is further configured to assign the weight to each of the plurality of captured images such that the weight becomes smaller as a distance from the image capturing unit that has captured the captured image to a reference viewpoint serving as a viewpoint of the synthesized image increases. ### Claim 6 (depends on 1) The apparatus according to claim 1, wherein the selected image is captured by the image capturing unit located at a position symmetrical about a reference viewpoint serving as a viewpoint of the synthesized image to a position of the image capturing unit that has captured the defective image. ### Claim 7 (depends on 6) The apparatus according to claim 6, wherein the selected image is captured by the image capturing unit located at a position line-symmetrical about a symmetry axis passing through the reference viewpoint to a position of the image capturing unit that has captured the defective image. ### Claim 8 (depends on 1) The apparatus according to claim 1, wherein said determination unit is further configured to determine the weight of the selected image by referring to a database. ### Claim 9 (depends on 1) The apparatus according to claim 1, wherein the selected image is selected such that a distance from the gravity center of the image capturing units each of which has captured the defective image or the selected image to a reference viewpoint serving as a viewpoint of the synthesized image is less than a predetermined threshold. ### Claim 10 (depends on 1) The apparatus according to claim 1, wherein said determination unit is further configured to set the gravity center position of the image capturing units that have not captured the defective image or the selected image as the reference viewpoint according to the determined weights. ### Claim 11 (depends on 1) The apparatus according to claim 1, wherein the plurality of image capturing units are arranged on grid points. ### Claim 12 (depends on 1) The apparatus according to claim 1, wherein the plurality of image capturing units are arranged on concentric circles. ### Claim 13 (depends on 1) The apparatus according to claim 1, further comprising a calculation unit configured to calculate a shift amount to align each of the plurality of captured images with an image obtained by image capturing from a reference viewpoint, wherein said synthesis unit is further configured to shift each of the plurality of captured images based on the shift amount calculated by said calculation unit and synthesize the shifted images. ### Claim 14 (depends on 1) An image capturing apparatus comprising: an image processing apparatus of claim 1; and a plurality of image capturing units configured to capture an object from different viewpoints. ### Claim 15 (depends on 1) A non-transitory computer-readable storage medium storing a computer program for causing a computer to function as each unit of the image processing apparatus according to claim 1. ### Claim 16 (depends on 1) The apparatus according to claim 1, wherein the determination unit is further configured to determine the weights so that the gravity center position of the image capturing units that have not captured the defective image or the selected image is equal to a predetermined position of the reference viewpoint. ### Claim 17 An image processing method comprising the steps of: acquiring captured images captured by a plurality of image capturing units for capturing an object from different viewpoints; specifying a defective image from the plurality of captured images; determining a weight for each captured image based on a position of the image capturing unit that has captured the defective image specified in said specifying step; and generating a synthesized image by weighted synthesis of the plurality of captured images based on the weights determined in said determining step, wherein, in the determination step, at least one captured image different from the defective image is selected as a selected image from the plurality of captured images, and the weights for each of the defective image and the selected image are determined to be smaller than a weight of a captured image that is neither the defective image nor the selected image, wherein, in the determination step, a relationship between the weights and a reference viewpoint serving as a viewpoint of the synthesized image is determined so that a gravity center of the image capturing units that have not captured the defective image or the selected image is set as the reference viewpoint. ### Claim 18 An image processing apparatus comprising: an acquisition unit configured to acquire captured images captured by a plurality of image capturing units for capturing an object from different viewpoints; a specifying unit configured to specify a defective image from the plurality of captured images; a determination unit configured to determine a weight for each captured image based on a position of the image capturing unit that has captured the defective image specified by said specifying unit; and a synthesis unit configured to generate a synthesized image by weighted synthesis of the plurality of captured images based on the weights determined by said determination unit, wherein said determination unit is further configured to select at least one captured image different from the defective image as a selected image from the plurality of captured images, and determine the weights for each of the defective image and the selected image to be smaller than a weight of a captured image that is neither the defective image nor the selected image, wherein a reference viewpoint serving as a viewpoint of the synthesized image is placed at a gravity center of the image capturing units that have not captured the defective image or the selected image. ### Claim 19 An image processing method comprising the steps of: acquiring captured images captured by a plurality of image capturing units for capturing an object from different viewpoints; specifying a defective image from the plurality of captured images; determining a weight for each captured image based on a position of the image capturing unit that has captured the defective image specified in said specifying step; and generating a synthesized image by weighted synthesis of the plurality of captured images based on the weights determined in said determining step, wherein, in the determining step, at least one captured image different from the defective image is selected as a selected image from the plurality of captured images, and the weights for each of the defective image and the selected image to be smaller than a weight of a captured image that is neither the defective image nor the selected image, wherein a reference viewpoint serving as a viewpoint of the synthesized image is placed at a gravity center of the image capturing units that have not captured the defective image or the selected image.