# US 8,150,201 B2 · Image processing apparatus, method, and computer program with pixel brightness-change detection and value correction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,150,201 B2 - Document ID: US8150201B2 - Status: Granted - Kind: B2 - Application number: 12/041,876 - Filed: 2008-03-04 - Published: 2008-09-11 - Granted: 2012-04-03 - Earliest claimed: 2007-03-09 - Assignees: Sony Corporation - Inventors: Masanori Kasai; Yusuke Oike - Web page: https://patentgrep.com/patent/US8150201B2 ## CPC - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 25/133: including elements passing panchromatic light, e.g. filters passing white light - H04N 25/589: with different integration times, e.g. short and long exposures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing apparatus includes the following elements. An image input unit receives a long-time exposure image and a short-time exposure image. An image analysis unit detects a brightness-change pixel in which a brightness change has occurred during a photographic period on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image. A pixel value correction unit corrects a pixel value of the detected brightness-change pixel. In the pixel value correction unit, a combined image generator selectively combines pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image; an intermediate image generator generates a blurred image of the combined image; and an output image generator determines a pixel value of the detected brightness-change pixel using a pixel value of a corresponding pixel in each of the combined image and the blurred image. ## Claims ### Claim 1 An image processing apparatus comprising: a processor configured to: receive input image data comprising a long-time exposure image and a short-time exposure image; detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and correct a pixel value of the brightness-change pixel detected by the image analysis unit, wherein the pixel value correction includes: selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image, generating an intermediate image, the intermediate image being a blurred image of the combined image, and determining a pixel value of the brightness-change pixel using a pixel value of a corresponding pixel in the combined image and a pixel value of a corresponding pixel in the intermediate image, the corresponding pixels in the combined and intermediate images being located at positions corresponding to the brightness-change pixel. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to determine a pixel value of the corresponding pixel in an output image using the following equation: [Rv]=a×[Dv]+b×[Mv] where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denote the pixel value of the corresponding pixel in the intermediate image, and a and b denote coefficients, wherein a+b=1. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to detect a pixel position at which an effective output value is obtained from the short-time exposure image and at which an effective output value is obtained from the long-time exposure image, and is configured to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a high brightness level to a low brightness level. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to detect a pixel position at which a pixel value in the long-time exposure image is saturated and at which no effective output value is obtained from the short-time exposure image, and is configured to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a low brightness level to a high brightness level. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to perform image conversion on a detection-result image that is based on the detected brightness-change pixel using a smoothing filter, and is configured to exclude a brightness-change pixel that is erroneously detected on the basis of an image obtained as a result of the image conversion. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to perform image conversion on a detection-result image that is based on the detected brightness-change pixel using a filter, and is configured to enlarge a brightness-change pixel region. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the processor is configured to determine a pixel value of the corresponding pixel in an output image using coefficients by the following equation: [Rv]=a×[Dv]+b×[Mv] where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denotes the pixel value of the corresponding pixel in the intermediate image, and a and b denote the coefficients, wherein a+b=1, and wherein the processor is further configured to determine the pixel value of the corresponding pixel in the output image by using, as the coefficients, a first set of coefficients used to determine pixel values of a brightness-change pixel region that is detected on the basis of the combined image, and a second set of coefficients to determine pixel values of the brightness-change pixel region enlarged using the filter, the first set of coefficients and the second set of coefficients having different values from each other. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to apply a smoothing filter to the combined image to generate a blurred image. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is configured to reduce and enlarge the combined image to generate a blurred image. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to reduce a color saturation of the combined image generated to generate a saturation-reduced image, and wherein the processor is further configured to receive the saturation-reduced image and to generate a blurred image. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to perform an image correction process of applying a low pass filter. ### Claim 12 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11) The image processing apparatus according to any of claims 1 to 11, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and wherein the short-time exposure image and the long-time exposure image are images generated from an identical pixel by a single imaging element. ### Claim 13 An imaging apparatus comprising: an imaging device configured to generate, by a processor, image data comprising a long-time exposure image and a short-time exposure image, wherein the processor is configured to: detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and correct a pixel value of the brightness-change pixel detected by the image analysis unit, wherein the correcting a pixel value includes: selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image, generating an intermediate image, the intermediate image being a blurred image of the combined image, and determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of a corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel. ### Claim 14 (depends on 13) The imaging apparatus according to claim 13, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and wherein the imaging device is configured to generate the short-time exposure image and the long-time exposure image from an identical pixel using a single imaging element. ### Claim 15 An image processing method for performing image processing in an image processing apparatus, the image processing method comprising the steps of: by an image input unit, receiving a long-time exposure image and a short-time exposure image; by an image analysis unit, detecting a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and by a pixel value correction unit, correcting a pixel value of the detected brightness-change pixel, wherein the step of correcting includes the steps of generating a combined image by selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image, generating an intermediate image, the intermediate image being a blurred image of the combined image, and determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of the corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel. ### Claim 16 (depends on 15) The image processing method according to claim 15, wherein the step of determining includes the step of: determining a pixel value of the corresponding pixel in an output image using the following equation: [Rv]=a×[Dv]+b×[Mv] where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denote the pixel value of the corresponding pixel in the intermediate image, and a and b denote coefficients, wherein a+b=1. ### Claim 17 (depends on 15) The image processing method according to claim 15, wherein the step of detecting includes the step of: detecting a pixel position at which an effective output value is obtained from the short-time exposure image and at which an effective output value is obtained from the long-time exposure image to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a high brightness level to a low brightness level. ### Claim 18 (depends on 15) The image processing method according to claim 15, wherein the step of detecting includes the step of: detecting a pixel position at which a pixel value in the long-time exposure image is saturated and at which no effective output value is obtained from the short-time exposure image to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a low brightness level to a high brightness level. ### Claim 19 (depends on 15) The image processing method according to claim 15, wherein the step of detecting includes the step of: performing image conversion on a detection-result image that is based on the detected brightness-change pixel using a smoothing filter to exclude a brightness-change pixel that is erroneously detected on the basis of an image obtained as a result of the image conversion. ### Claim 20 (depends on 15) The image processing method according to claim 15, wherein the step of detecting includes the step of: performing image conversion on a detection-result image that is based on the detected brightness-change pixel using a filter to enlarge a brightness-change pixel region. ### Claim 21 (depends on 20) The image processing method according to claim 20, wherein the step of determining includes the step of: determining a pixel value of the corresponding pixel in an output image using coefficients by the following equation: [Rv]=a×[Dv]+b×[Mv] where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denotes the pixel value of the corresponding pixel in the intermediate image, and a and b denote the coefficients, wherein a+b=1, wherein the pixel value of the corresponding pixel in the output image is determined using, as the coefficients, a first set of coefficients used to determine pixel values of a brightness-change pixel region that is detected on the basis of the combined image, and a second set of coefficients to determine pixel values of the brightness-change pixel region enlarged using the filter, the first set of coefficients and the second set of coefficients having different values from each other. ### Claim 22 (depends on 15) The image processing method according to claim 15, wherein the step of generating an intermediate image includes the step of: applying a smoothing filter to the combined image to generate a blurred image. ### Claim 23 (depends on 15) The image processing method according to claim 15, wherein the step of generating an intermediate image includes the step of: reducing and enlarging the combined image to generate a blurred image. ### Claim 24 (depends on 15) The image processing method according to claim 15, wherein the step of correcting further includes the step of: reducing a color saturation of the generated combined image to generate a saturation-reduced image, and wherein the step of generating an intermediate image generates, upon receiving the generated saturation-reduced image, a blurred image. ### Claim 25 (depends on 15) The image processing method according to claim 15, wherein the step of correcting further includes the step of: performing an image correction process of applying a low pass filter to an image generated in the step of determining. ### Claim 26 (depends on 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25) The image processing method according to any of claims 15 to 25, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and wherein the short-time exposure image and the long-time exposure image are images generated from an identical pixel by a single imaging element. ### Claim 27 A non-transitory computer storage medium comprising a computer program for causing an image processing apparatus to execute image processing, the computer program comprising the steps of: causing an image input unit to receive a long-time exposure image and a short-time exposure image; causing an image analysis unit to detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and causing a pixel value correction unit to correct a pixel value of the brightness-change pixel detected in the step of causing an image analysis unit to detect a brightness-change pixel, wherein the step of causing a pixel value correction unit to correct a pixel value includes the steps of generating a combined image by selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image, generating a blurred image of the combined image, and generating an output image by determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of the corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel.