# US 11,443,410 B2 · Image processing method and image processing apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,410 B2 - Document ID: US11443410B2 - Status: Granted - Kind: B2 - Application number: 17/061,845 - Filed: 2020-10-02 - Published: 2021-04-15 - Granted: 2022-09-13 - Earliest claimed: 2019-10-10 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Hiroto Mizohana; Keita Dan - Web page: https://patentgrep.com/patent/US11443410B2 ## CPC - G06T 7/593: from stereo images - G06T 3/4069: by subpixel displacements - G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images - G06T 2207/10012: Stereo images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing method includes a matching cost calculating process of calculating matching costs in a unit of sub-pixels having higher resolution than first and second images by using an image of a reference area contained in the first image in which a target object is imaged and images of a plurality of comparison areas contained in the second image in which the target object is imaged, and a synthesized cost calculating process of calculating synthesized costs related to the reference area based on comparison results of values related to the plurality of matching costs calculated in the matching cost calculating process. ## Claims ### Claim 1 An image processing method comprising: a matching cost calculating process of calculating matching costs in a unit of sub-pixels having higher resolution than first and second images by using (1) an image of a reference area based on the first image in which a target object is imaged and (2) images of a plurality of comparison areas based on the second image in which the target object is imaged; and a synthesized cost calculating process of calculating synthesized costs related to the reference area based on comparison results of values related to the plurality of matching costs calculated in the matching cost calculating process, wherein the matching costs in the unit of sub-pixels are calculated based on a comparison of the reference area contained in a first high resolution image obtained by enhancing resolution of the first image with the plurality of comparison areas contained in a second high resolution image obtained by enhancing resolution of the second image in the matching cost calculating process, or wherein the matching costs in the unit of sub-pixels are calculated based on matching costs in a unit of pixels calculated by comparing the reference area contained in the first image with the plurality of comparison areas contained in the second image in the matching cost calculating step, by using a correlation function between parallax amounts and the matching costs related to the reference area and estimated based on the matching costs in the unit of pixels calculated by comparing the reference area contained in the first image and the plurality of comparison areas contained in the second image. ### Claim 2 (depends on 1) The image processing method according to claim 1, further comprising a process of calculating a parallax amount of the target object imaged in the reference area based on the synthesized costs. ### Claim 3 (depends on 1) The image processing method according to claim 1, wherein, in a case where the correlation function is used, the correlation function is estimated by fitting a quadratic function of the parallax amounts to the calculated matching costs in the unit of sub-pixels in the matching cost calculating process. ### Claim 4 An image processing method, comprising: a matching cost calculating process of calculating matching costs in a unit of sub-pixels having higher resolution than first and second images by using (1) an image of a reference area based on the first image in which a target object is imaged and (2) images of a plurality of comparison areas based on the second image in which the target object is imaged; and a synthesized cost calculating process of calculating synthesized costs related to the reference area based on comparison results of values related to the plurality of matching costs calculated in the matching cost calculating process, wherein the synthesized costs are calculated by acting a penalty cost which corresponds to the size of the reference area in the synthesized cost calculating process. ### Claim 5 (depends on 1) A non-transitory computer-readable recording storage medium storing a control program for making a control unit execute each process of the image processing method according to claim 1. ### Claim 6 (depends on 4) A non-transitory computer-readable recording storage medium configured to store a control program for making a control unit execute each process of the image processing method according to claim 4. ### Claim 7 An image processing apparatus comprising: an image acquisition portion configured to acquire first and second images in which a target object is imaged; a cost map generating portion configured to calculate matching costs in a unit of sub-pixels having higher resolution than the first and second images by using (1) an image of a reference area based on the first image and (2) images of a plurality of comparison areas based on the second image; and a synthesized cost calculating portion configured to calculate synthesized costs related to the reference area based on comparison results of values related to the plurality of matching costs calculated by the cost map generating portion, wherein the cost map generating portion calculates the matching costs in the unit of sub-pixels based on a comparison of the reference area contained in a first high resolution image obtained by enhancing resolution of the first image with the plurality of comparison areas contained in a second high resolution image obtained by enhancing resolution of the second image, or wherein the cost map generating portion calculates matching costs in the unit of sub-pixels based on matching costs in a unit of pixels calculated by comparing the reference area contained in the first image with the plurality of comparison areas contained in the second image, by using a correlation function between parallax amounts and the matching costs related to the reference area and estimated based on the calculated matching costs in the unit of pixels. ### Claim 8 (depends on 7) The image processing apparatus according to claim 7, further comprising a parallax amount generating portion configured to calculate a parallax amount of the target object imaged in the reference area based on the synthesized costs. ### Claim 9 (depends on 7) The image processing apparatus according to claim 7, wherein, in a case where the correlation function is used, the correlation function is estimated by fitting a quadratic function of the parallax amounts to the matching costs in the unit of sub-pixels. ### Claim 10 An image processing apparatus comprising: an image acquisition portion configured to acquire first and second images in which a target object is imaged; a cost map generating portion configured to calculate matching costs in a unit of sub-pixels having higher resolution than the first and second images by using (1) an image of a reference area contained in the first image and (2) images of a plurality of comparison areas contained in the second image; and a synthesized cost calculating portion configured to calculate synthesized costs related to the reference area based on comparison results of values related to the plurality of matching costs calculated by the cost map generating portion, wherein the synthesized cost generating portion calculates the synthesized cost by acting a penalty cost which corresponding to the size of the reference area. ### Claim 11 (depends on 7) The image processing apparatus according to claim 7, further comprising a user interface configured to receive a parameter from the user related to a parallax amount calculation unit of the matching costs in the unit of sub-pixels.