# US 11,909,942 B2 · Parallax detecting apparatus, image capturing apparatus, parallax detecting method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,909,942 B2 - Document ID: US11909942B2 - Status: Granted - Kind: B2 - Application number: 17/082,321 - Filed: 2020-10-28 - Published: 2021-05-13 - Granted: 2024-02-20 - Earliest claimed: 2019-11-11 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Shin Tanaka - Web page: https://patentgrep.com/patent/US11909942B2 ## CPC - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - G01C 3/085: {with electronic parallax measurement} - H04N 13/257: Colour aspects - H04N 23/63: by using electronic viewfinders - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) ## Abstract There is provided a parallax detecting apparatus. A first obtainment unit obtains a first viewpoint image and a second viewpoint image that have different viewpoints from each other in a first direction. A second obtainment unit obtains displacement information that specifies a displacement amount in a second direction between positions in the first viewpoint image and the second viewpoint image corresponding to a same position in a shooting range. The displacement amount varies in conformity with a parallax amount between the positions. The second direction is perpendicular to the first direction. A calculation unit performs correlation calculation processing with respect to each of a plurality of reference regions corresponding to a plurality of reference positions in the second viewpoint image. A detection unit detects a parallax amount at a base position in the first viewpoint image based on calculated correlations. ## Claims ### Claim 1 A parallax detecting apparatus comprising at least one processor and/or at least one circuit which functions as: a first obtainment unit configured to obtain a first viewpoint image and a second viewpoint image that have different viewpoints from each other in a first direction; a second obtainment unit configured to obtain displacement information that specifies a displacement amount in a second direction between positions in the first viewpoint image and the second viewpoint image corresponding to a same position in a shooting range, wherein the displacement amount varies in conformity with a parallax amount in the first direction between the positions, the parallax amount being in conformity with a defocus amount, and wherein the second direction is perpendicular to the first direction; a selection unit configured to select, in the first viewpoint image, a base region corresponding to a base position of parallax amount detection; a calculation unit configured to perform correlation calculation processing with respect to each of a plurality of reference regions corresponding to a plurality of reference positions in the second viewpoint image, wherein the correlation calculation processing includes first correction processing and calculation processing, wherein the first correction processing corrects at least one of a position of the base region in the second direction and a position of the reference region in the second direction based on the displacement information so as to reduce a displacement that occurs between the base position and the reference position in the second direction and that is in conformity with a parallax amount in the first direction between the base position and the reference position, and wherein the calculation processing calculates a correlation between the base region and the reference region that reflect the first correction processing; and a detection unit configured to detect a parallax amount in the first direction at the base position based on each correlation between the base region and each of the plurality of reference regions. ### Claim 2 (depends on 1) The parallax detecting apparatus according to claim 1, wherein in the first correction processing, when a post-correction pixel position of the base region or the reference region that serves as a correction target region is displaced from a pixel position of the first viewpoint image or the second viewpoint image that includes the correction target region, the calculation unit calculates a post-correction pixel value of the correction target region through interpolation computation. ### Claim 3 (depends on 2) The parallax detecting apparatus according to claim 2, wherein the interpolation computation is based on a plurality of pixel values that are continuous in the second direction. ### Claim 4 (depends on 3) The parallax detecting apparatus according to claim 3, wherein the interpolation computation is linear interpolation computation. ### Claim 5 (depends on 2) The parallax detecting apparatus according to claim 2, wherein the interpolation computation is linear interpolation computation based on two pixel values that are continuous in the second direction, wherein the at least one processor and/or at least one circuit further functions as a control unit configured to, with respect to at least one of the first viewpoint image and the second viewpoint image, perform control to calculate, with respect to each of a plurality of combinations of two pixel values that are continuous in the second direction, a pixel difference value representing a difference between the two pixel values and record the pixel difference value into a recording unit, and wherein in the first correction processing, when linear interpolation computation based on two pixel values corresponding to the pixel difference value recorded in the recording unit is to be performed, the calculation unit performs the linear interpolation computation using the pixel difference value. ### Claim 6 (depends on 1) The parallax detecting apparatus according to claim 1, wherein the at least one processor and/or at least one circuit further functions as a correction unit configured to perform second correction processing for correcting at least one of a position of the first viewpoint image in the second direction and a position of the second viewpoint image in the second direction based on the displacement information so as to reduce a displacement in the second direction that occurs between positions of the first viewpoint image and the second viewpoint image corresponding to a same position in the shooting range in a case where a parallax amount in the first direction between the positions is a specific parallax amount, and wherein the calculation unit determines a correction amount for the first correction processing based on the displacement information and a correction amount for the second correction processing. ### Claim 7 (depends on 6) The parallax detecting apparatus according to claim 6, wherein the specific parallax amount is 0. ### Claim 8 (depends on 6) The parallax detecting apparatus according to claim 6, wherein the correction unit determines the specific parallax amount based on a frequency of occurrence of a parallax amount in the first direction in the shooting range. ### Claim 9 (depends on 6) The parallax detecting apparatus according to claim 6, wherein in the second correction processing, when a post-correction pixel position of the first viewpoint image or the second viewpoint image that serves as a correction target image is displaced from a pre-correction pixel position of the correction target image, the correction unit calculates a post-correction pixel value of the correction target image through interpolation computation. ### Claim 10 (depends on 6) The parallax detecting apparatus according to claim 6, wherein the correction unit determines a correction amount for the first viewpoint image and a correction amount for the second viewpoint image in the second correction processing so as to achieve a displacement amount of 0 in the second direction between the positions of the first viewpoint image and the second viewpoint image corresponding to the same position in the shooting range in the case where the parallax amount in the first direction between the positions is the specific parallax amount. ### Claim 11 (depends on 10) The parallax detecting apparatus according to claim 10, wherein the correction unit determines the correction amount for the first viewpoint image and the correction amount for the second viewpoint image in the second correction processing based on a pixel arrangement of an image sensor that has been used in capturing the first viewpoint image and the second viewpoint image, so as to reduce a difference between a correction error in the first viewpoint image and a correction error in the second viewpoint image. ### Claim 12 (depends on 1) The parallax detecting apparatus according to claim 1, wherein in the calculation processing, the calculation unit calculates the correlation between the base region and the reference region based on an SAD (Sum of Absolute Differences), an SSD (Sum of Squared Differences), or a normalized cross correlation. ### Claim 13 (depends on 1) The parallax detecting apparatus according to claim 1, wherein the displacement information includes information indicating an inclination and an intercept in a graph with a horizontal axis representing a parallax amount in the first direction and a vertical axis representing a displacement amount. ### Claim 14 (depends on 1) The parallax detecting apparatus according to claim 1, wherein the second obtainment unit specifies the displacement amount that varies in conformity with a parallax amount in the first direction and obtains the displacement information by performing a two-dimensional search between the first viewpoint image and the second viewpoint image. ### Claim 15 (depends on 14) The parallax detecting apparatus according to claim 14, wherein the second obtainment unit updates the displacement information at a predetermined timing. ### Claim 16 (depends on 1) The parallax detecting apparatus according to claim 1, wherein the displacement information includes information indicating an inclination and an intercept in a graph with a horizontal axis representing a parallax amount in the first direction and a vertical axis representing a displacement amount, wherein the second obtainment unit specifies the displacement amount that varies in conformity with a parallax amount in the first direction and obtains the displacement information by performing a two-dimensional search between the first viewpoint image and the second viewpoint image, and wherein the second obtainment unit updates the intercept of the displacement information at a predetermined timing. ### Claim 17 (depends on 1) The parallax detecting apparatus according to claim 1, wherein the detection unit calculates a subject distance at the base position based on the detected parallax amount in the first direction. ### Claim 18 (depends on 1) An image capturing apparatus comprising: the parallax detecting apparatus according to claim 1, and an image sensor configured to generate the first viewpoint image and the second viewpoint image. ### Claim 19 A parallax detecting method executed by a parallax detecting apparatus, the method comprising: obtaining a first viewpoint image and a second viewpoint image that have different viewpoints from each other in a first direction; obtaining displacement information that specifies a displacement amount in a second direction between positions in the first viewpoint image and the second viewpoint image corresponding to a same position in a shooting range, wherein the displacement amount varies in conformity with a parallax amount in the first direction between the positions, the parallax amount being in conformity with a defocus amount, and wherein the second direction is perpendicular to the first direction; selecting, in the first viewpoint image, a base region corresponding to a base position of parallax amount detection; performing correlation calculation processing with respect to each of a plurality of reference regions corresponding to a plurality of reference positions in the second viewpoint image, wherein the correlation calculation processing includes first correction processing and calculation processing, wherein the first correction processing corrects at least one of a position of the base region in the second direction and a position of the reference region in the second direction based on the displacement information so as to reduce a displacement that occurs between the base position and the reference position in the second direction and that is in conformity with a parallax amount in the first direction between the base position and the reference position, and wherein the calculation processing calculates a correlation between the base region and the reference region that reflect the first correction processing; and detecting a parallax amount in the first direction at the base position based on each correlation between the base region and each of the plurality of reference regions. ### Claim 20 A non-transitory computer-readable storage medium which stores a program for causing a computer to execute a parallax detecting method, the method comprising: obtaining a first viewpoint image and a second viewpoint image that have different viewpoints from each other in a first direction; obtaining displacement information that specifies a displacement amount in a second direction between positions in the first viewpoint image and the second viewpoint image corresponding to a same position in a shooting range, wherein the displacement amount varies in conformity with a parallax amount in the first direction between the positions, the parallax amount being in conformity with a defocus amount, and wherein the second direction is perpendicular to the first direction; selecting, in the first viewpoint image, a base region corresponding to a base position of parallax amount detection; performing correlation calculation processing with respect to each of a plurality of reference regions corresponding to a plurality of reference positions in the second viewpoint image, wherein the correlation calculation processing includes first correction processing and calculation processing, wherein the first correction processing corrects at least one of a position of the base region in the second direction and a position of the reference region in the second direction based on the displacement information so as to reduce a displacement that occurs between the base position and the reference position in the second direction and that is in conformity with a parallax amount in the first direction between the base position and the reference position, and wherein the calculation processing calculates a correlation between the base region and the reference region that reflect the first correction processing; and detecting a parallax amount in the first direction at the base position based on each correlation between the base region and each of the plurality of reference regions.