# US 7,746,387 B2 · Methods and systems for synthesizing pickup images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 7,746,387 B2 - Document ID: US7746387B2 - Status: Granted - Kind: B2 - Application number: 11/724,253 - Filed: 2007-03-15 - Published: 2007-08-30 - Granted: 2010-06-29 - Earliest claimed: 1998-12-21 - Assignees: Sony Corporation - Inventors: Masami Ogata; Takashi Tsuchiya; Kazuhiko Ueda - Web page: https://patentgrep.com/patent/US7746387B2 ## CPC - H04N 25/41: {Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors} - H04N 23/73: by influencing the exposure time - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 25/583: with different integration times - H04N 25/589: with different integration times, e.g. short and long exposures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In an image pickup apparatus, a level compensator compensates the level of an image xL (i, j) sensed by an image sensor by exposure for a long time on the basis of the exposure of the image to produce a compensated image xL′ (i, j), while another level compensator compensates the level of an image xS (i, j) sensed by the image sensor by exposure for a short time on the basis of the exposure of the image to produce a compensated image xS′ (i, j). The image pickup apparatus also includes an image synthesizer to synthesize these compensates images xL′ (i, j) and xS′ (i, j) to produce a single synthetic image x (i, j). A dynamic range compressor also included in the image pickup apparatus compresses the synthetic image x (i, j) to produce a compressed image y (i, j) for delivery to outside. ## Claims ### Claim 1 An image pickup apparatus adapted to produce a single image from a plurality of images acquired by sensing an object under different exposure conditions, comprising: means for sensing the object under different exposure conditions to acquire the plurality of images; means for calculating a pixel value for each of the plurality of images; means for multiplying the pixel value by a factor set based on the respective exposure condition for the image to calculate a positive value compensation amount; means for synthesizing the plurality of compensated images to produce a single synthetic image having a wide dynamic range; means for compressing the dynamic range of the synthesized image to an extent depending upon the performance of an output destination of the image pickup apparatus to produce a compressed image; first means for compensating the level of the plurality of images on the basis of the respective exposure conditions under which the plurality of images have been sensed by subtracting a positive compensation value from each of the plurality of images, to produce a plurality of compensated images; means for calculating the positive compensation value by multiplying a level average of each of the plurality of images by a respective coefficient for the image; and means for selecting the coefficient for each of the plurality of images based on the respective exposure condition of that image. ### Claim 2 (depends on 1) The image pickup apparatus of claim 1, further comprising second means for compensating the compressed image by subtracting a positive value from each pixel level of the compressed image to produce a compensated compressed image. ### Claim 3 (depends on 2) The image pickup apparatus of claim 2, the second means for compensating further comprising: means for low-pass filtering a signal of the compressed image; means for normalizing the filtered signal; and means for multiplying the normalized signal by a predetermined factor to calculate the positive value. ### Claim 4 (depends on 3) The image pickup apparatus of claim 3, the second means for compensating further comprising: means for time-smoothing the positive value; and means for subtracting the time-smoothed positive value from each pixel level of the compressed image. ### Claim 5 (depends on 1) The image pickup apparatus of claim 1, wherein: each of the plurality of images is an image signal in which a frequency-modulated color signal is superposed on a brightness signal; the image pickup apparatus further comprises means for separating the image signal into the brightness and color signals; the separated brightness signal and the separated color signal are synthesized by the means for synthesizing to produce a synthetic brightness signal and a synthetic color signal, respectively; the synthetic brightness and color signals are compressed by the means for compressing to produce a compressed brightness signal and a compressed color signal, respectively; and the compressed brightness and color signals are compensated by the means for second compensating means to produce a compensated compressed brightness signal and a compensated compressed color signal, respectively. ### Claim 6 (depends on 5) The image pickup apparatus of claim 5, wherein the second means for compensating comprises: means for calculating a compensation amount based on the compressed brightness signal; means for compensating the level of the compressed brightness signal with the compensation amount to produce a compensated compressed brightness signal; and means for producing the compensated compressed color signal based on the compensated compressed brightness signal. ### Claim 7 (depends on 5) The image pickup apparatus of claim 5, further comprising means for mixing the compensated compressed brightness signal and the compensated compressed color signal. ### Claim 8 An image pickup apparatus adapted to synthesize a plurality of images acquired by sensing an object under different exposure conditions to produce a single image, the apparatus comprising: means for sensing the object under different exposure conditions to acquire the plurality of images; means for calculating a pixel value for each of the plurality of images; means for multiplying the pixel value by a factor set based on the respective exposure condition for the image to calculate a positive value compensation amount for the image; first means for compensating the levels of the plurality of images on the basis of the respective exposure conditions under which the plurality of images have been sensed by subtracting a positive compensation value from each of the plurality of images; means for calculating the positive compensation value by multiplying a level average of each of the plurality of images by a respective coefficient for the image; and means for selecting the coefficient for each of the plurality of images based on the exposure condition of that image; means for synthesizing the plurality of compensated images to produce a single synthesized image having a wide dynamic range; means for compressing the dynamic range of the synthesized image to an extent depending upon the performance of an output destination of the image pick up apparatus to produce a compressed image; and second means for compensating the level of the compressed image to provide a compensated compressed image. ### Claim 9 (depends on 8) The apparatus as set forth in claim 8, wherein the second compensating means comprises means for subtracting a predetermined positive value from each pixel level of the compressed image to produce the compensated compressed image. ### Claim 10 (depends on 9) The apparatus as set forth in claim 9, wherein the second compensating means further comprises: means for filtering a signal of the compressed image; means for normalizing the filtered signal; and means for multiplying the normalized signal by a predetermined factor to calculate a positive value. ### Claim 11 (depends on 10) The apparatus as set forth in claim 10, wherein the second compensating means further comprises; means for time-smoothing the positive value; and means for subtracting the time-smoothed positive value from each pixel level of the compressed image. ### Claim 12 (depends on 8) The apparatus as set forth in claim 8, wherein: the means for sensing the object is adapted to output an image signal in which a frequency-modulated color signal is superposed on a brightness signal, the apparatus further comprises means for separating the image signal into the brightness and color signals; the means for synthesizing synthesizes the separated brightness and color signals to produce a synthesized brightness signal and a synthesized color signal, respectively; the means for compressing compresses the synthesized brightness and color signals to produce a compressed brightness signal and a compressed color signal, respectively; and the second means for compensating compensates the compressed brightness and color signals to produce a compensated compressed brightness signal and a compensated compressed color signal, respectively. ### Claim 13 (depends on 12) The apparatus as set forth in claim 12, wherein the level second compensating means further comprises: means for calculating a compensation amount based on the separated compressed brightness signal third means for compensating the level of the compressed brightness signal with the compensation amount to produce a compensated compressed brightness signal, and means for producing the compensated compressed color signal based on the compensated compressed brightness signal. ### Claim 14 (depends on 12) The apparatus as set forth in claim 12, further comprising means for mixing the compensated compressed brightness and color signals. ### Claim 15 An image processing method of synthesizing a plurality of input images acquired by sensing an object under different exposure conditions to produce a single image, the method comprising the steps of: receiving a plurality of images acquired by sensing an object under different exposure conditions; providing a plurality of compensated images by subtracting a positive compensation value from each of the plurality of images; wherein the positive compensation value is calculated by multiplying a level average of each of the plurality of images by a coefficient based on the respective exposure condition for the image; wherein the coefficient is selected for each of the plurality of images based on the exposure condition of that image; calculating a pixel value for each of the plurality of images; multiplying the pixel value by a factor set based on the respective exposure condition for the image to calculate the positive compensation value; synthesizing the plurality of input images to produce a single synthetic image having a wide dynamic range; compressing the dynamic range of the synthetic image to an extent depending upon the performance of an output destination to produce a compressed image; and compensating the level of the compressed image to provide a compensated compressed image. ### Claim 16 (depends on 15) The method as set forth in claim 15, wherein compensating the level of the compressed image further comprises the step of subtracting a predetermined positive value from each pixel level of the compressed image to produce the compensated compressed image. ### Claim 17 (depends on 16) The method as set forth in claim 16, wherein calculating the positive compensation value further comprises the steps of: low-pass filtering the signal of the compressed image; normalizing the filtered signal; and multiplying the normalized signal by a predetermined factor to calculate the positive value. ### Claim 18 (depends on 17) The method as set forth in claim 17, further comprising the steps of: time-smoothing the positive value obtained; and wherein providing a plurality of compensated images includes subtracting the time-smoothed positive value from each pixel level of the compressed image. ### Claim 19 (depends on 15) The method as set forth in claim 15, wherein: each of the plurality of input images is an image signal in which a frequency-modulated color signal is superposed on a brightness signal, and the method further comprises: separating the image signal into the brightness and color signals; and synthesizing the separated brightness signal and the separated color signal to produce a synthesized brightness signal and a synthesized color signal, respectively; compressing the synthesized brightness and color signals to produce a compressed brightness signal and a compressed color signal, respectively; and compensating the compressed brightness and color signals to produce a compensated compressed brightness signal and a compensated compressed color signal, respectively. ### Claim 20 (depends on 19) The method as set forth in claim 19, wherein compensating the level of the compressed image includes: calculating a compensation amount based on the separated compressed brightness signal; compensating the level of the compressed brightness signal to produce a compensated compressed brightness signal; and producing the compensated compressed color signal based on the compensated compressed brightness signal. ### Claim 21 (depends on 19) The method as set forth in claim 19, further comprising the step of mixing the compensated compressed brightness and color signals. ### Claim 22 An image processing apparatus adapted to synthesize a plurality of input images acquired by sensing an object under different exposure conditions to produce a single image, the apparatus comprising: means for receiving a plurality of images acquired by sensing an object under different exposure conditions; first means for compensating the levels of the plurality of input images on the basis of the respective exposure conditions under which the plurality of images have been sensed; means for providing a plurality of compensated images by subtracting a positive compensation from each of the plurality of images; means for calculating said positive compensation value by multiplying a level average of each of the plurality of images by a coefficient based on the corresponding exposure condition for the image; means for selecting said coefficient for each of the plurality of images based on the respective exposure condition of that image; means for calculating a pixel value for each of the plurality of images; means for multiplying the pixel value by a factor set based on the respective exposure condition for the image to calculate the positive compensation value; means for synthesizing the plurality of images to produce a single synthesized image having a wide dynamic range; means for compressing the dynamic range of the synthesized image to an extent depending upon the performance of an output destination of the image processing apparatus to produce a compressed image; and second means for compensating the level of the compressed image to provide a compensated compressed image. ### Claim 23 (depends on 22) The apparatus as set forth in claim 22, wherein the second compensating means comprises means for subtracting a positive value from each pixel level of the compressed image to produce the compensated compressed image. ### Claim 24 (depends on 23) The apparatus as set forth in claim 23, wherein the second compensating means further comprises: means for filtering a signal of the compressed image; means for normalizing the filtered signal; and means for multiplying the normalized signal by a predetermined factor to calculate the positive value. ### Claim 25 (depends on 24) The apparatus as set forth in claim 24, wherein the second compensating means further comprises: means for time-smoothing the positive value; and means for subtracting the time-smoothed positive value from each pixel level of the compressed image. ### Claim 26 (depends on 22) The apparatus as set forth in claim 22, further comprising means for separating the image signal into the brightness and color signals; wherein the signal synthesizing means synthesizes the separated brightness signal and the separated color signal to produce a synthesized brightness signal and a synthesized color signal, respectively; the means for compressing compresses the synthesized brightness and color signals to produce a compressed brightness signal and a compressed color signal, respectively; and the second compensating means compensates the compressed brightness and color signals to produce a compensated compressed brightness signal and a compensated compressed color signal, respectively. ### Claim 27 (depends on 26) The apparatus as set forth in claim 26, wherein the second compensating means comprises: means for calculating a compensation amount based on the compressed brightness signal separated by the signal separating means; means for compensating the level of the compressed brightness signal with the compensation amount to produce a compensated compressed brightness signal; and means for producing the compensated compressed color signal based on the compensated compressed brightness signal. ### Claim 28 (depends on 26) The apparatus as set forth in claim 26, further comprising means for mixing the compensated compressed brightness and color signals.