# US 10,455,149 B2 · Image processing apparatus, image processing apparatus control method, image pickup apparatus, and image pickup apparatus control method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,455,149 B2 - Document ID: US10455149B2 - Status: Granted - Kind: B2 - Application number: 15/872,465 - Filed: 2018-01-16 - Published: 2018-05-24 - Granted: 2019-10-22 - Earliest claimed: 2014-08-20 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Fumitaka Nakayama - Web page: https://patentgrep.com/patent/US10455149B2 ## CPC - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - G06T 7/571: from focus - H04N 13/128: Adjusting depth or disparity - H04N 23/672: {based on the phase difference signals} - H04N 23/95: Computational photography systems, e.g. light-field imaging systems - G06T 2207/10148: Varying focus - H04N 25/704: Pixels specially adapted for focusing, e.g. phase difference pixel sets - H04N 2013/0092: {Image segmentation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing apparatus including an acquiring unit configured to acquire an image signal of a plurality of images obtained by picking up an object image that is formed by a photographing optical system; a determining unit configured to determine a defocus amount of the object image by using the image signal of the plurality of images; and a conversion unit configured to perform, on the defocus amount, gradation conversion that has at least one conversion characteristic out of a plurality of different conversion characteristics, and to output information that is based on the defocus amount converted by the gradation conversion. ## Claims ### Claim 1 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that more gradations are allocated to distance information near an in-focus region as compared with distance information not near the in-focus region, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from a plurality of conversion characteristics, and in case where an application for which distance resolution near the in-focus region is necessary is a destination of the output, the conversion unit is configured to determine the conversion characteristic as a conversion characteristic that more resolutions are allocated to the distance information near the in-focus region as compared with the distance information not near the in-focus region. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and there is further provided a conversion characteristic that the gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and there is further provided a conversion characteristic that the resolution conversion varies depending on an F-value of a lens of the photographing optical system. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and there is further provided a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, the gradations to be allocated to larger distance information are made fewer. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and there is further provided a conversion characteristic that the resolution conversion varies depending on a base length in the pickup of the object image. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and there is further provided a conversion characteristic that the resolution conversion varies depending on an ISO sensitivity in the pickup of the object image. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and in a case where an application for which a relatively wider distance measurement range is necessary is a destination of the output, the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that liner resolution conversion is to be performed. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the distance information is information of an object distance, the determining unit further comprises a generating unit configured to generate, as the distance information, the information of the object distance from a defocus amount acquired based on the image signals of the plurality of images, and the conversion unit is configured to convert the object distance depending on the conversion characteristic. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the conversion unit is configured to determine one or more conversion characteristics respectively for the distance information, convert the distance information depending on the respective conversion characteristics, and output a plurality of results acquired by the conversion. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the image signals of the plurality of images include image signals of a plurality of images of the object image different from one another in blurring, image signals of a plurality of images acquired by picking up the object image from different regions of a pupil of the photographing optical system, or image signals of a plurality of images acquired by picking up an object with a plurality of image pickup units. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the distance information is based on a value indicating a degree of match between the plurality of images or a blurring evaluation value of the plurality of images. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein the distance information is information which is calculated from the image signals of the plurality of images and corresponds to an object distance of an object in the image. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein distance information converted using the conversion characteristic by the conversion unit is output as an object distance map. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the distance information is information representing a defocus amount calculated from the image signals of the plurality of images. ### Claim 15 (depends on 1) The image processing apparatus according to claim 1, wherein the distance information converted using the conversion characteristic by the conversion unit is output as a defocus map. ### Claim 16 (depends on 1) The image processing apparatus according to claim 1, further comprising an image processing unit configured to perform an image process using distance information converted by the conversion unit. ### Claim 17 (depends on 16) The image processing apparatus according to claim 16, wherein the image process includes image deformation process of deforming, by using distance information converted by the conversion unit, each region of the image depending on distance information. ### Claim 18 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object. ### Claim 19 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on an F-value of a lens of the photographing optical system. ### Claim 20 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, gradations to be allocated to larger distance information are made fewer. ### Claim 21 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on a base length in the pickup of the object image. ### Claim 22 An image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as: an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion, wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on an ISO sensitivity in the pickup of the object image. ### Claim 23 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that more gradations are allocated to distance information near an in-focus region as compared with distance information not near the in focus region, wherein performing the resolution conversion includes selectively determining the conversion characteristic from a plurality of conversion characteristics, and in a case where an application for which distance resolution near the in-focus region is necessary is a destination of the output, determining the conversion characteristic as a conversion characteristic that more resolutions are allocated to the distance information near the in-focus region as compared with the distance information not near the in-focus region. ### Claim 24 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object. ### Claim 25 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on an F-value of a lens of the photographing optical system. ### Claim 26 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, gradations to be allocated to larger distance information are made fewer. ### Claim 27 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on a base length in the pickup of the object image. ### Claim 28 An image processing method comprising: acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system; determining distance information of the object image by using the image signals of the plurality of images; and performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion, wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on an ISO sensitivity in the pickup of the object image.