# US 2021/0243379 A1 · Image processing device, control method thereof, imaging device, and recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0243379 A1 - Document ID: US20210243379A1 - Status: Published application - Kind: A1 - Application number: 17/162,855 - Filed: 2021-01-29 - Published: 2021-08-05 - Earliest claimed: 2020-02-03 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Kazuma Sakato; Yohei Horikawa - Web page: https://patentgrep.com/patent/US20210243379A1 ## CPC - H04N 23/672: {based on the phase difference signals} - H04N 23/675: {comprising setting of focusing regions} - G06T 5/40: using histogram techniques - H04N 23/617: Upgrading or updating of programs or applications for camera control - H04N 23/62: Control of parameters via user interfaces - H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters} - H04N 23/635: {Region indicators; Field of view indicators} - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An imaging device acquires a plurality of viewpoint image signals having different viewpoints from an imaging element. The imaging device selects a subject region within a captured image, and divides the subject region into a plurality of regions. The imaging device performs a correlation operation for each divided region, and calculates an amount of parallax on the basis of one or more extreme positions. In addition, the imaging device calculates distance information of a subject on the basis of the amount of parallax, and calculates a reliability for the distance information. If there are two or more pieces of distance information which are output in the divided region, the imaging device generates a distance histogram using the two or more pieces of distance information based on the reliability ## Claims ### Claim 1 An image processing device comprising at least one processor and a memory holding a program which makes the processor function as: an acquisition unit configured to acquire a plurality of image signals having different viewpoints; a division unit configured to divide a region within an image into a plurality of regions; a parallax amount calculation unit configured to calculate an amount of parallax by performing a correlation operation of the plurality of image signals with respect to a plurality of divided regions divided by the division unit; a distance calculation unit configured to calculate distance information of a subject based on the amount of parallax; a reliability calculation unit configured to calculate a reliability which is an index for the distance information in the correlation operation; and a generation unit configured to generate a histogram representing a frequency distribution of the distance information, wherein, if a plurality of pieces of the distance information are present in the divided regions, the generation unit generates the histogram using the plurality of pieces of distance information and the reliability. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the generation unit generates the histogram by performing a weighting operation using the reliability and accumulating a frequency of the distance information corresponding to a result of the weighting operation. ### Claim 3 (depends on 2) The image processing device according to claim 2, wherein the generation unit adds the distance information when the reliability is larger than a first threshold to a frequency of the histogram, or does not add the distance information when the reliability is smaller than a second threshold to the frequency of the histogram. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the reliability is a degree of steepness of a change in an amount of correlation in the correlation operation or an extreme value which is obtained by the correlation operation. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the processor further functions as a selection unit configured to select a subject region within an image, and wherein the division unit divides the selected subject region into a plurality of regions. ### Claim 6 (depends on 5) The image processing device according to claim 5, wherein, if a size of the subject region is smaller than a third threshold, the generation unit does not generate the histogram. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein the processor further functions as: a display unit configured to display the histogram generated by the generation unit; and a designation unit configured to designate a position corresponding to a subject to be focused on the histogram displayed by the display unit. ### Claim 8 (depends on 7) The image processing device according to claim 7, wherein the display unit displays the histogram at a position different from a subject region within a displayed image. ### Claim 9 (depends on 8) The image processing device according to claim 8, wherein, if a size of the subject region is larger than a fourth threshold, the display unit does not display the histogram. ### Claim 10 (depends on 7) The image processing device according to claim 7, wherein the display unit identifiably displays a region out of a plurality of the divided regions in which the distance information corresponding to a position of the histogram designated by the designation unit is calculated. ### Claim 11 An imaging device comprising: an imaging optical system; an imaging element; at least one processor and a memory holding a program which makes the processor function as an acquisition unit configured to acquire a plurality of image signals having different viewpoints, a division unit configured to divide a region within an image into a plurality of regions, a parallax amount calculation unit configured to calculate an amount of parallax by performing a correlation operation of the plurality of image signals with respect to a plurality of divided regions divided by the division unit, a distance calculation unit configured to calculate distance information of a subject based on the amount of parallax, a reliability calculation unit configured to calculate a reliability which is an index for the distance information in the correlation operation, a generation unit configured to generate a histogram representing a frequency distribution of the distance information, and a control unit configured to perform focus adjustment control on the imaging optical system using focus detection information acquired by the parallax amount calculation unit, wherein, if a plurality of pieces of the distance information are present in the divided regions, the generation unit generates the histogram using the plurality of pieces of distance information and the reliability. ### Claim 12 (depends on 11) The imaging device according to claim 11, wherein the imaging element includes a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each of the microlenses, and the acquisition unit acquires the plurality of image signals from the plurality of photoelectric conversion units. ### Claim 13 A control method which is executed in an image processing device that acquires and processes a plurality of image signals having different viewpoints, the method comprising: dividing a region within an image into a plurality of regions; calculating an amount of parallax by performing a correlation operation of the plurality of image signals with respect to a plurality of divided regions divided in the dividing; calculating distance information of a subject based on the amount of parallax; calculating a reliability which is an index for the distance information in the correlation operation; and generating a histogram representing a frequency distribution of the distance information, wherein the generating includes, if a plurality of pieces of the distance information are present in the divided region, generating the histogram using the plurality of pieces of distance information and the reliability. ### Claim 14 A non-transitory recording medium storing a control program of an image processing device that acquires and processes a plurality of image signals having different viewpoints causing a computer to perform each step of a control method of the image processing device, the method comprising: dividing a region within an image into a plurality of regions; calculating an amount of parallax by performing a correlation operation of the plurality of image signals with respect to a plurality of divided regions divided in the dividing; calculating distance information of a subject based on the amount of parallax; calculating a reliability which is an index for the distance information in the correlation operation; and generating a histogram representing a frequency distribution of the distance information, wherein the generating includes, if a plurality of pieces of the distance information are present in the divided region, generating the histogram using the plurality of pieces of distance information and the reliability.