# US 8,149,283 B2 · Image processing device, electronic camera, image processing method, and image processing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,149,283 B2 - Document ID: US8149283B2 - Status: Granted - Kind: B2 - Application number: 12/312,983 - Filed: 2008-01-17 - Published: 2010-02-04 - Granted: 2012-04-03 - Earliest claimed: 2007-01-23 - Assignees: Nikon Corporation - Inventors: Shigeru Doida - Web page: https://patentgrep.com/patent/US8149283B2 ## CPC - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - H04N 1/32128: {attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image (for information embedded in the image H04N1/32144)} - H04N 1/3871: {the composed originals being of different kinds, e.g. low- and high-resolution originals} - H04N 1/407: Control or modification of tonal gradation or of extreme levels, e.g. background level - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - H04N 23/6811: {based on the image signal} - H04N 23/70: Circuitry for compensating brightness variation in the scene - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 2207/10144: Varying exposure - H04N 2101/00: Still video cameras - H04N 2201/0084: Digital still camera - H04N 2201/3257: relating to the original, the input device or the input process, e.g. scanner profile - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract To provide a technique to obtain a synthesized image with an expanded dynamic range with ease and a technique to reduce workload of a user while maintaining the degree of freedom relating to image processing. There provides an image inputting part taking in at least one of a plurality of low-resolution images and a high-resolution image obtained by shooting the same subject while changing exposure condition, a shift detecting part detecting a positional shift of pictorial pattern between each of the plurality of low-resolution images and the high-resolution image, and a gradation expanding part generating a synthesized image in which a range of reproduced gradation is expanded by performing position alignment between each of the plurality of low-resolution images and the high-resolution image based on the positional shift, extracting a gradation information of the plurality of low-resolution images, and synthesizing it with the high-resolution image. ## Claims ### Claim 1 An image processing device, comprising: an image inputting part taking in at least one of a plurality of low-resolution images and a high-resolution image obtained by shooting the same subject while changing exposure condition; a shift detecting part detecting a positional shift of pictorial pattern between each of the plurality of low-resolution images and the high-resolution image; and a gradation expanding part generating a synthesized image in which a range of reproduced gradation is expanded by performing position alignment between each of the plurality of low-resolution images and the high-resolution image based on the positional shift, extracting gradation information of the plurality of low-resolution images, and synthesizing the plurality of low-resolution images with the high-resolution image. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the image inputting part takes in two or more low-resolution images; and the gradation expanding part synthesizes the gradation information of low-resolution represented by each of the plurality of low-resolution images with the high-resolution image in a multiplexing manner. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the shift detecting part comprises: a phase dividing part generating a plurality of pieces of sampling information with sample positions shifted from one another by extracting an edge component of pictorial pattern from the high-resolution image and performing sub-sampling of the edge component while shifting phases; and a precisely detecting part detecting a positional shift with a finer precision than a pixel interval of the low-resolution images by detecting the positional shift with which the pictorial pattern best matches between each of the plurality of low-resolution images and the plurality of pieces of sampling information. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the gradation expanding part: determines a high-brightness region and a low-brightness region of the high-resolution image; increases a synthesis ratio of one or more low-resolution images underexposed as to the high-brightness region of the high-resolution image; and increases a synthesis ratio of one or more low-resolution images overexposed as to the low-brightness region of the high-resolution image. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the gradation expanding part comprises: an adjusting unit adjusting a tone level of a corresponding pixel of a synthesized image by reflecting a tone level of each pixel included in at least one of the low-resolution images chosen from the plurality of low-resolution images and a tone level of corresponding pixel of the high-resolution image; and a controlling unit controlling adjusting processing of the tone level of each pixel of the synthesized image by the adjusting unit based on at least one of the followings, a histogram of the tone level obtained for each of the plurality of low-resolution images, a histogram of the tone level obtained for the high-resolution image, and the number of the low-resolution images acquired by the image inputting part. ### Claim 6 (depends on 5) The image processing device according to claim 5, wherein the controlling unit comprises: an analyzing unit analyzing a distribution of pixels in a predetermined range of tone levels as to the histogram of the tone level obtained for each of the plurality of low-resolution images and the histogram of the tone level of the high-resolution image; and a converting curve fitting unit adjusting a gradation converting curve used to confine the tone level of each pixel of the synthesized image within a predetermined range in the adjusting unit based on the analysis result by the analyzing unit. ### Claim 7 (depends on 5) The image processing device according to claim 5, wherein the controlling unit comprises: an analyzing unit analyzing a distribution of pixels in a predetermined range of tone levels as to the histogram of the tone level obtained for each of the plurality of low-resolution images and the histogram of the tone level of the high-resolution image; and a choosing unit choosing at least one of the low-resolution images to be used in adjusting processing by the adjusting unit based on the analysis result by the analyzing unit. ### Claim 8 (depends on 5) The image processing device according to claim 5, wherein the controlling unit comprises a range determining unit determining the size of a region of low-resolution images to be reflected in the adjustment of the tone level of each pixel included in the synthesized image by the adjusting unit in accordance with the number of low-resolution images to be used in synthesizing processing by the adjusting unit. ### Claim 9 (depends on 5) The image processing device according to claim 5, wherein the controlling unit comprises: a brightness weight determining unit determining a brightness weight to be applied when reflecting a brightness component, which corresponds to the pixel of at least one of the low-resolution images chosen, to a brightness component of each pixel included in the synthesized image by the adjusting unit; and a color-difference weight determining unit determining a color-difference weight to be applied when reflecting a color-difference component, which corresponds to the pixel of at least one of the low-resolution image chosen, to a color-difference component of each pixel included in the synthesized image by the adjusting unit, wherein the color-difference weight determining unit comprises a weight adjusting unit adjusting a value of the color-difference weight in accordance with the magnitude of a color-difference component corresponding to each pixel of the high-resolution image. ### Claim 10 (depends on 1) An electronic camera, comprising: the image processing device according to claim 1; and an imaging part shooting a subject with at least two kinds of resolution, wherein the high-resolution image processed by the image processing device is a still image of the high-resolution shot by the imaging part, and the image processing device processes at least one of the plurality of low-resolution images shot by the imaging part before and/or after the shooting of the still image with the exposure condition different from the high-resolution image. ### Claim 11 (depends on 10) The electronic camera according to claim 10, comprising a monitor part displaying an image, wherein the imaging part: sequentially shoots low-resolution through images, through images are images obtained by pixel skipping to provide moving images for view finder, and displays a moving image on the monitor part; and shoots a plurality of low-resolution images with timing not synchronizing the shooting of the through images and under the exposure condition different from the high-resolution image. ### Claim 12 (depends on 1) A non-transitory computer readable medium storing an image processing program capable of instructing a computer to function as the image processing device according to claim 1. ### Claim 13 An image processing method, comprising: an image inputting step for acquiring a plurality of low-resolution images obtained by shooting the same subject under a plurality of exposure conditions with an imaging unit shooting images of the subject with a plurality of different resolutions and a high-resolution image obtained by shooting the subject under a correct exposure condition with the imaging unit; and a synthesizing step for synthesizing at least one of the low-resolution images chosen from the plurality of low-resolution images and the high-resolution image, and generating a synthesized image having a resolution equivalent to the high-resolution image, wherein the synthesizing step comprises: an adjusting step for adjusting a tone level of a corresponding pixel of the synthesized image by reflecting a tone level of each pixel included in at least one of the low-resolution images chosen and a tone level of corresponding pixel of the high-resolution image; and a controlling step for controlling adjusting processing of the tone level of each pixel of the synthesized image in the adjusting step based on at least one of the followings, a histogram of a tone level obtained for each of the plurality of low-resolution images, a histogram of a tone level obtained for the high-resolution image, and the number of the low-resolution images acquired in the image inputting step.