# US 11,445,109 B2 · Image processing device, image capturing device, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,109 B2 - Document ID: US11445109B2 - Status: Granted - Kind: B2 - Application number: 16/715,583 - Filed: 2019-12-16 - Published: 2020-04-16 - Granted: 2022-09-13 - Earliest claimed: 2017-07-05 - Assignees: Olympus Corporation - Applicants: Olympus Corporation - Inventors: Eiji Furukawa; Shunichi Koga - Web page: https://patentgrep.com/patent/US11445109B2 ## CPC - G06T 3/4069: by subpixel displacements - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 23/12: with one sensor only - H04N 25/48: Increasing resolution by shifting the sensor relative to the scene - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing device comprises: one or more processors comprising hardware, the one or more processors being configured to: generate a high-resolution combined image by aligning a plurality of time-series images with each other in a high-resolution image space having a resolution higher than the plurality of time-series images based on an amount of displacement between the plurality of time-series images, and combining the plurality of time-series images; generate a low-resolution combined image by aligning the plurality of time-series images with each other in a low-resolution image space having a resolution equal to or lower than the resolution of the plurality of time-series images based on the amount of displacement, combining the plurality of time-series images through weighted addition; calculate a feature quantity pertaining to a pixel-value change direction at each region in the generated low-resolution combined image; and correct the high-resolution combined image based on the calculated feature quantity. ## Claims ### Claim 1 An image processing device comprising: at least one processor comprising hardware, the at least one processor being configured to: generate a high-resolution combined image by aligning a plurality of time-series images with each other in a high-resolution image space having a resolution higher than a resolution of the plurality of time-series images based on an amount of displacement between the plurality of time-series images, and combining the plurality of time-series images; generate a low-resolution combined image by aligning the plurality of time-series images with each other in a low-resolution image space having a resolution equal to or lower than the resolution of the plurality of time-series images based on the amount of displacement, and combining the plurality of time-series images through weighted addition; calculate a feature quantity pertaining to a pixel-value change direction at each of plural regions in the generated low-resolution combined image; and correct the high-resolution combined image based on the calculated feature quantity. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to detect an amount of motion between the plurality of time-series images as the amount of displacement between the plurality of time-series images. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the generating of the high-resolution combined image comprises arranging each pixel of the plurality of time-series images in the high-resolution image space based on the amount of displacement. ### Claim 4 (depends on 3) The image processing device according to claim 3, wherein the generating of the high-resolution combined image comprises interpolating a pixel that is not arranged in the high-resolution space based on information about pixels arranged in the high-resolution image space. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the generating of the low-resolution combined image comprises arranging each pixel of the plurality of time-series images in the low-resolution image space based on the amount of displacement. ### Claim 6 (depends on 5) The image processing device according to claim 5, wherein the generating of the low-resolution combined image comprises interpolating a pixel that is not arranged in the low-resolution space based on information about pixels arranged in the low-resolution image space. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein the calculating of the feature quantity comprises calculating, as the feature quantity, an edge direction at each of the plural regions in the low-resolution combined image. ### Claim 8 (depends on 7) The image processing device according to claim 7, wherein the calculating of the feature quantity comprises extending a resolution of a distribution of the feature quantities to a resolution of the high-resolution combined image. ### Claim 9 (depends on 7) The image processing device according to claim 7, wherein the correcting of the high-resolution combined image comprises applying, to the high-resolution combined image, filtering along the calculated edge direction. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the calculating of the feature quantity comprises calculating, as the feature quantity, a correlation between each of the plural regions and a peripheral region thereof in the low-resolution combined image. ### Claim 11 (depends on 10) The image processing device according to claim 10, wherein the calculating of the feature quantity comprises extending a resolution of a distribution of the feature quantities to a resolution of the high-resolution combined image. ### Claim 12 (depends on 10) The image processing device according to claim 10, wherein the correcting of the high-resolution combined image comprises applying bilateral filtering to the high-resolution combined image by using a weighting coefficient that is set based on a distance between each of the plural regions and the peripheral region thereof and correcting the weighting coefficient so that the higher the correlation, the larger the weighting coefficient and so that the lower the correlation, the smaller the weighting coefficient. ### Claim 13 (depends on 1) An image capturing device comprising: an image acquisition unit that is configured to acquire a plurality of time-series images; and the image processing device according to claim 1 for processing the acquired plurality of time-series images. ### Claim 14 (depends on 13) The image capturing device according to claim 13, wherein the image acquisition unit includes: an image capturing element; a sensor shift mechanism that is configured to shift a position of the image capturing element in a pixel arrangement direction of the image capturing element; and a sensor shift controller that is configured to control a direction of shift and an amount of shift of the image capturing element performed by the sensor shift mechanism. ### Claim 15 (depends on 14) The image capturing device according to claim 14, wherein the at least one processor is configured to calculate the amount of displacement based on the direction of shift and the amount of shift of the image capturing element controlled by the sensor shift control unit. ### Claim 16 An image processing method comprising: generating, by at least one processor comprising hardware, a high-resolution combined image by aligning a plurality of time-series images with each other in a high-resolution image space having a resolution higher than a resolution of the plurality of time-series images based on an amount of displacement between the plurality of time-series images, and combining the plurality of time-series images; generating, by the at least one processor, a low-resolution combined image by aligning the plurality of time-series images with each other in a low-resolution image space having a resolution equal to or lower than the resolution of the plurality of time-series images based on the amount of displacement, and combining the plurality of time-series images through weighted addition; calculating, by the at least one processor, a feature quantity pertaining to a pixel-value change direction at each of plural regions in the generated low-resolution combined image; and correcting, by the at least one processor, the high-resolution combined image based on the calculated feature quantity. ### Claim 17 A non-transitory computer-readable storage medium storing an image processing program thereon, the program being executable to control a computer to execute processes comprising: generating a high-resolution combined image by aligning the plurality of time-series images with each other in a high-resolution image space having a resolution higher than a resolution of the plurality of time-series images based on an amount of displacement between the plurality of time-series images, and combining the plurality of time-series images; generating a low-resolution combined image by aligning the plurality of time-series images with each other in a low-resolution image space having a resolution equal to or lower than the resolution of the plurality of time-series images based on the amount of displacement, and combining the plurality of time-series images through weighted addition; calculating a feature quantity pertaining to a pixel-value change direction at each of plural regions in the generated low-resolution combined image; and correcting the high-resolution combined image based on the calculated feature quantity.