# US 2010/0165129 A1 · Image processing apparatus, image processing method and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2010/0165129 A1 - Document ID: US20100165129A1 - Status: Published application - Kind: A1 - Application number: 12/720,367 - Filed: 2010-03-09 - Published: 2010-07-01 - Earliest claimed: 2004-02-13 - Assignees: Sony Corporation - Inventors: Tetsujiro KONDO; Masanori KANEMARU - Web page: https://patentgrep.com/patent/US20100165129A1 ## CPC - H04N 23/682: {Vibration or motion blur correction} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/73: Deblurring; Sharpening - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - H04N 23/6811: {based on the image signal} - G06T 2207/10016: Video; Image sequence - G06T 2207/20201: Motion blur correction - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An apparatus for processing an image includes a motion vector detector, a motion-blurring-mitigated image generator, and a spatial resolution creation unit. The motion vector detector is configured to detect a motion vector by using an image that is made up of multiple pixels and acquired by an image sensor having time integration effects. The motion-blurring-mitigated image generator is configured to generate a motion-blurring-mitigated image in which motion blurring of a moving object is mitigated by using the motion vector detected by the motion vector detector on the assumption that a pixel value of pixel of the moving object in the image is a value obtained by integrating, in a time direction, a pixel value of each pixel in which no motion blurring that corresponds to the moving object occur as it is moved. The spatial resolution creation unit is configured to generate an image that has a higher spatial resolution than that of the image made up of the multiple pixels by using the motion vector detected by the motion vector detector and the motion-blurring-mitigated image from the motion-blurring-mitigated image generator. ## Claims ### Claim 1 An apparatus for processing an image, said apparatus comprising: motion vector detection means for detecting a motion vector by using an image that is made up of multiple pixels and acquired by an image sensor having time integration effects; motion-blurring-mitigated image generation means for generating a motion-blurring-mitigated image in which motion blurring of a moving object is mitigated by using the motion vector detected by the motion vector detection means on the assumption that a pixel value of pixel of the moving object in the image is a value obtained by integrating, in a time direction, a pixel value of each pixel in which no motion blurring that corresponds to the moving object occur as it is moved; and spatial resolution creation means for generating an image that has a higher spatial resolution than that of the image made up of the multiple pixels by using the motion vector detected by the motion vector detection means and the motion-blurring-mitigated image from the motion-blurring-mitigated image generation means. ### Claim 2 (depends on 1) The apparatus for processing the image according to claim 1, wherein the motion vector detection means detects a motion vector by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor and, based on the detected motion vector, generates a motion vector for the image having the higher spatial resolution to supply this motion vector to the spatial resolution creation means. ### Claim 3 (depends on 1) The apparatus for processing the image according to claim 1, wherein the motion vector detection means detects a motion vector by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor, corrects the detected motion vector in accordance with exposure lapse of time, and supplies it to the motion-blurring-mitigated image generation means. ### Claim 4 (depends on 1) The apparatus for processing the image according to claim 1, wherein the spatial resolution creation means uses the motion-blurring-mitigated image to generate an image having a higher spatial resolution than that of the motion-blurring-mitigated image. ### Claim 5 (depends on 4) The apparatus for processing the image according to claim 4, wherein the spatial resolution creation means comprises: class determination means for determining a motion vector of a target pixel in the image, to be created, with the higher spatial resolution, by using the motion vector detected by the motion vector detection means, extracting as a class tap multiple pixels that correspond to the target pixel from the motion-blurring-mitigated image, and determining a class that corresponds to the target pixel based on a pixel value of the class tap; storage means for storing prediction coefficients for predicting a target pixel based on multiple pixels in a first image for each class; prediction value generation means for detecting from the storage means a prediction coefficient that corresponds to a class determined by the class determination means, extracting as a prediction tap multiple pixels that correspond to the target pixel in the image to be generated from the motion-blurring-mitigated image, and generating interpolated pixel data corresponding to the target pixel according to one-dimensional linear combination of the predictive coefficients detected from the storage means and the predictive tap; expanded image output means for generating the image that has the higher spatial resolution than that of the image made up of the multiple pixels based on the notion vector detected by the notion vector detection means and the interpolated pixel data. ### Claim 6 (depends on 1) The apparatus for processing the image according to claim 1, wherein the spatial resolution creation means generates the image that has image that has the higher spatial resolution than that of the image made up of the multiple pixels such that the image that has image that has the higher spatial resolution has twice as many vertical pixels and twice as many horizontal pixels as the image made up of the multiple pixels. ### Claim 7 A method for processing an image, said method comprising: detecting a notion vector by using an image made up of multiple pixels acquired by an image sensor having time integration effects; generating a notion-blurring-mitigated image in which notion blurring of a moving object is mitigated by using the notion vector detected by the detecting on the assumption that a pixel value of pixel of the moving object in the image is a value obtained by integrating, in a time direction, a pixel value of each pixel in which no notion blurring that corresponds to the moving object occur as it is moved; and generating an image that has a higher spatial resolution than that of the image made up of the multiple pixels by using the notion vector detected by the detecting a notion vector and the notion-blurring-mitigated image from the generating a notion-blurring-mitigated image. ### Claim 8 (depends on 7) The method for processing the image according to claim 7, wherein in the detecting a motion vector, a notion vector by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor is detected and, based on the detected notion vector, a notion vector for the image having the higher spatial resolution is generated and supplied to the generating an image. ### Claim 9 (depends on 7) The method for processing the image according to claim 7, wherein in the detecting a notion vector, a notion vector is detected by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor, and the detected notion vector is corrected in accordance with exposure lapse of time and supplied to the generating a notion-blurring-mitigated image. ### Claim 10 (depends on 7) The method for processing the image according to claim 7, wherein in the generating an image, the notion-blurring-mitigated image is used to generate an image having a higher spatial resolution than that of the motion-blurring-mitigated image. ### Claim 11 (depends on 10) The method for processing the image according to claim 10, wherein the generating an image comprises: determining a motion vector of a target pixel in the image, to be created, with the higher spatial resolution, by using the motion vector detected by detecting a motion vector, extracting as a class tap multiple pixels that correspond to the target pixel from the notion-blurring-mitigated image, and determining a class that corresponds to the target pixel based on a pixel value of the class tap; storing prediction coefficients for predicting a target pixel based on multiple pixels in a first image for each class; detecting a prediction coefficient that corresponds to a class determined by the determining a class, extracting as a prediction tap multiple pixels that correspond to the target pixel in the image to be generated from the motion-blurring-mitigated image, and generating interpolated pixel data corresponding to the target pixel according to one-dimensional linear combination of the predictive coefficients and the predictive tap; generating the image that has the higher spatial resolution than that of the image made up of the multiple pixels based on the notion vector detected by the detecting a notion vector and the interpolated pixel data. ### Claim 12 (depends on 7) The method for processing the image according to claim 7, wherein generating an image generates the image that has image that has the higher spatial resolution than that of the image made up of the multiple pixels such that the image that has image that has the higher spatial resolution has twice as many vertical pixels and twice as many horizontal pixels as the image made up of the multiple pixels. ### Claim 13 A computer readable medium including computer executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform a method comprising: detecting a notion vector by using an image made up of multiple pixels acquired by an image sensor having time integration effects; generating a notion-blurring-mitigated image in which motion blurring of a moving object is mitigated by using the motion vector detected by the detecting on the assumption that a pixel value of pixel of the moving object in the image is a value obtained by integrating, in a time direction, a pixel value of each pixel in which no motion blurring that corresponds to the moving object occur as it is moved; and generating an image that has a higher spatial resolution than that of the image made up of the multiple pixels by using the motion vector detected by the detecting a motion vector and the motion-blurring-mitigated image from the generating a motion-blurring-mitigated image. ### Claim 14 An apparatus for processing an image, said apparatus comprising: a motion vector detector configured to detect a motion vector by using an image that is made up of multiple pixels and acquired by an image sensor having time integration effects; a motion-blurring-mitigated image generator configured to generate a motion-blurring-mitigated image in which motion blurring of a moving object is mitigated by using the motion vector detected by the motion vector detector on the assumption that a pixel value of pixel of the moving object in the image is a value obtained by integrating, in a time direction, a pixel value of each pixel in which no motion blurring that corresponds to the moving object occur as it is moved; and a spatial resolution creation unit configured to generate an image that has a higher spatial resolution than that of the image made up of the multiple pixels by using the motion vector detected by the motion vector detector and the motion-blurring-mitigated image from the motion-blurring-mitigated image generator. ### Claim 15 (depends on 14) The apparatus for processing the image according to claim 14, wherein the motion vector detector is configured to detect a motion vector by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor and, based on the detected motion vector, generates a motion vector for the image having the higher spatial resolution to supply this motion vector to the spatial resolution creation unit. ### Claim 16 (depends on 14) The apparatus for processing the image according to claim 14, wherein the motion vector detector is configured to detect a motion vector by using multiple images, each of which is made up of multiple pixels, acquired by the image sensor, corrects the detected motion vector in accordance with exposure lapse of time, and supplies it to the motion-blurring-mitigated image generator. ### Claim 17 (depends on 14) The apparatus for processing the image according to claim 14, wherein the spatial resolution creation unit uses the motion-blurring-mitigated image to generate an image having a higher spatial resolution than that of the motion-blurring-mitigated image. ### Claim 18 (depends on 17) The apparatus for processing the image according to claim 17, wherein the spatial resolution creation unit comprises: a class determination unit configured to determine a motion vector of a target pixel in the image, to be created, with the higher spatial resolution, by using the motion vector detected by the motion vector detector, extracting as a class tap multiple pixels that correspond to the target pixel from the motion-blurring-mitigated image, and determining a class that corresponds to the target pixel based on a pixel value of the class tap; a storage unit configured to store prediction coefficients for predicting a target pixel based on multiple pixels in a first image for each class; a prediction value generator configured to detect from the storage unit a prediction coefficient that corresponds to a class determined by the class determination unit, to extract as a prediction tap multiple pixels that correspond to the target pixel in the image to be generated from the motion-blurring-mitigated image, and to generate interpolated pixel data corresponding to the target pixel according to one-dimensional linear combination of the predictive coefficients detected from the storage unit and the predictive tap; an expanded image output unit configured to generate the image that has the higher spatial resolution than that of the image made up of the multiple pixels based on the motion vector detected by the motion vector detector and the interpolated pixel data. ### Claim 19 (depends on 14) The apparatus for processing the image according to claim 14, wherein the spatial resolution creation unit is configured to generate the image that has image that has the higher spatial resolution than that of the image made up of the multiple pixels such that the image that has image that has the higher spatial resolution has twice as many vertical pixels and twice as many horizontal pixels as the image made up of the multiple pixels.