# US 10,373,293 B2 · Image processing apparatus, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,293 B2 - Document ID: US10373293B2 - Status: Granted - Kind: B2 - Application number: 15/801,098 - Filed: 2017-11-01 - Published: 2018-05-10 - Granted: 2019-08-06 - Earliest claimed: 2016-11-04 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Akihito Taketani - Web page: https://patentgrep.com/patent/US10373293B2 ## CPC - G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/205: {Image-based rendering} - G06T 15/30: Clipping - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/0141: {characterised by the informative content of the display} - G06T 2207/10016: Video; Image sequence - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A first image and a second image with resolutions different from each other are generated from an image captured by an imaging unit, and the first image and the second image are subject to different types of image processing. A virtual image is generated by calculating a position and an orientation of the imaging unit based on the processed first image. A composite image is generated by combining the generated virtual image with the second image subjected to the image processing, and the generated composite image is used as a display image. ## Claims ### Claim 1 An image processing apparatus comprising: at least one processor; at least one memory; a first generation unit configured to generate a first image with a predetermined resolution and a second image with a resolution higher than the predetermined resolution, based on an image captured by an imaging unit; a first processing unit configured to perform first image processing on the generated first image; a second processing unit configured to perform second image processing, which is different from the first image processing, on the generated second image; a derivation unit configured to derive a position and a direction of the imaging unit by detecting a marker feature or a natural feature in the processed first image; a second generation unit configured to generate a virtual image, based on the derived position and direction; and a composition unit configured to generate a composite image based on the processed second image and the generated virtual image, wherein the first generation unit, the first processing unit, the derivation unit, the second generation unit, and the composition unit are realized by execution of a program stored in the at least one memory by the at least one processor. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the first image indicates a predetermined region, and the second image indicates a region corresponding to a part of the predetermined region. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the first generation unit generates the first image and the second image in such a manner that a total data amount of the first image and the second image is smaller than a data amount of the image captured by the imaging unit. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the first generation unit generates the first image by performing resolution conversion of the image captured by the imaging unit. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the first generation unit generates the second image by performing resolution conversion of the image captured by the imaging unit and clipping a part of the captured image. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the noise removal in the first image processing is emphasized in a space direction, and the noise removal in the second image processing is emphasized in a time direction. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the first image processing and the second image processing each include color correction and gamma correction, the color correction and the gamma correction in the first image processing emphasize a marker feature or a natural feature included in the captured image, and the color correction and the gamma correction in the second image processing provide high reproducibility of the captured image. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, further comprising a control unit configured to change the first image processing and the second image processing based on a change in a configuration of a system including the image processing apparatus, or a change in an external environment. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing apparatus is connected to an image display apparatus including the imaging unit and a display unit. ### Claim 10 (depends on 9) The image processing apparatus according to claim 9, wherein the image display apparatus is a head mounted display. ### Claim 11 An image processing method comprising: generating a first image with a predetermined resolution and a second image with a resolution higher than the predetermined resolution based on an image captured by an imaging unit; performing first image processing on the generated first image; performing second image processing, which is different from the first image processing, on the generated second image; deriving a position and a direction of the imaging unit based on the processed first image; generating a virtual image, based on the derived position and direction; and generating a composite image by deleting a marker feature or a natural feature in the processed second image and the generated virtual image. ### Claim 12 A non-transitory computer-readable recording medium that stores a program for causing a computer to function as the units of an image processing apparatus comprising: a first generation unit configured to generate a first image with a predetermined resolution and a second image with a resolution higher than the predetermined resolution based on an image captured by an imaging unit; a first processing unit configured to perform first image processing on the generated first image; a second processing unit configured to perform second image processing, which is different from the first image processing, on the generated second image; a derivation unit configured to derive a position and a direction of the imaging unit, by detecting a marker feature or a natural feature in the processed first image; a second generation unit configured to generate a virtual image based on the derived position and direction; and a composition unit configured generate a composite image, based on the processed second image and the generated virtual image.