# US 8,786,726 B2 · Imaging processing apparatus, camera system, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,786,726 B2 - Document ID: US8786726B2 - Status: Granted - Kind: B2 - Application number: 13/927,508 - Filed: 2013-06-26 - Published: 2013-10-31 - Granted: 2014-07-22 - Earliest claimed: 2010-05-31 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Shingo Nagataki; Kenji Wajima; Hiroshi Yamamoto; Katsuhisa Shinmei - Web page: https://patentgrep.com/patent/US8786726B2 ## CPC - H04N 23/73: by influencing the exposure time - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 25/589: with different integration times, e.g. short and long exposures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A camera system, image processing method and computer program stored on a computer readable medium for image processing. Long and short accumulation images with longer and shorter exposure times are respectively obtained from one field. First and second long accumulation evaluation values are computed from the long accumulation image. A short accumulation evaluation value is computed from the short accumulation image. A short accumulation target exposure time and short accumulation exposure control information are acquired from the short accumulation evaluation value and a first target level. A long accumulation evaluation value is acquired by synthesizing the first and second long accumulation evaluation values. Long accumulation exposure control information is acquired from the long accumulation evaluation value and a second target level. An image is generated from the long accumulation image and the short accumulation image. ## Claims ### Claim 1 A camera system comprising: an image processing apparatus including an image sensor; and an optical system that forms a subject image in the image sensor, wherein the image processing apparatus includes an image sensor that is exposed to light, images, and outputs at least two images of a long accumulation image with a longer exposure time and a short accumulation image with a shorter exposure time in one field, a first long accumulation evaluation value computation unit that computes an exposure evaluation value from the long accumulation image and outputs the value as a first long accumulation evaluation value, a second long accumulation evaluation value computation unit that computes an exposure evaluation value from the long accumulation image according to a different method from that of the first long accumulation evaluation value computation unit, and outputs the value as a second long accumulation evaluation value, a short accumulation evaluation value computation unit that computes an exposure evaluation value from the short accumulation image and outputs the value as a short accumulation evaluation value, a short accumulation exposure control unit that acquires a short accumulation target exposure time and short accumulation exposure control information from the short accumulation evaluation value and a first target level, a long accumulation evaluation value synthesis unit that acquires a long accumulation evaluation value by synthesizing the first long accumulation evaluation value and the second long accumulation evaluation value in response to the short accumulation target exposure time using a predetermined method, a long accumulation exposure control unit that acquires long accumulation exposure control information from the long accumulation evaluation value and a second target level, and an image sensor drive unit that drives the image sensor from the long accumulation exposure control information and the short accumulation exposure control information; a synthesis unit that generates an image from the long accumulation image and the short accumulation image in response to the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 2 (depends on 1) The camera system according to claim 1, further comprising an image sensor drive unit that drives the image sensor from the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 3 (depends on 1) The camera system according to claim 1, wherein the long accumulation exposure control information and the short accumulation exposure control information contain information of exposure times. ### Claim 4 (depends on 1) The camera system according to claim 1, wherein the first target level and the second target level take the same value. ### Claim 5 (depends on 1) The camera system according to claim 1, wherein the first long accumulation evaluation value computation unit computes the exposure evaluation value by putting a weight toward a low-brightness side, the second long accumulation evaluation value computation unit computes the exposure evaluation value by putting a weight toward a high-brightness side, and the short accumulation evaluation value computation unit computes the exposure evaluation value by putting a weight toward the high-brightness side, respectively. ### Claim 6 (depends on 5) The camera system according to claim 5, wherein the weights are equal in the second long accumulation evaluation value computation unit and the short accumulation evaluation value computation unit. ### Claim 7 (depends on 1) The camera system according to claim 1, wherein the long accumulation evaluation value synthesis unit takes the first long accumulation evaluation value if the short accumulation target exposure time is less than a first threshold value, and takes the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than the first threshold value as the long accumulation evaluation value, respectively. ### Claim 8 (depends on 7) The camera system according to claim 7, wherein the first threshold value is a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 9 (depends on 7) The camera system according to claim 7, wherein, if an amount of PGA gain in the short accumulation image is larger, the first threshold value is smaller. ### Claim 10 (depends on 1) The camera system according to claim 1, wherein the long accumulation evaluation value synthesis unit takes the first long accumulation evaluation value if the short accumulation target exposure time is less than a second threshold value, takes the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than a third threshold value larger than the second threshold value, and takes a value obtained by complementing the first long accumulation evaluation value and the second long accumulation evaluation value using a predetermined method if the short accumulation target exposure time is equal to or more than the second threshold value and less than the third threshold value as the long accumulation evaluation value, respectively. ### Claim 11 (depends on 10) The camera system according to claim 10, wherein the second threshold value takes a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 12 (depends on 10) The camera system according to claim 10, wherein, if an amount of PGA gain in the short accumulation image is larger, the second and third threshold values are smaller. ### Claim 13 (depends on 1) The camera system according to claim 1, wherein, if the short accumulation target exposure time is equal to or more than a fourth threshold value, in the synthesis unit, not the composite image of the long accumulation image and the short accumulation image, but the long accumulation image itself is output. ### Claim 14 (depends on 13) The camera system according to claim 13, wherein the fourth threshold value is equal to or more than the first threshold value. ### Claim 15 (depends on 13) The camera system according to claim 13, wherein the fourth threshold value is equal to or more than the third threshold value. ### Claim 16 An image processing method, the method comprising: imaging, within one field, at least a long accumulation image with a longer exposure time and a short accumulation image with a shorter exposure time; computing an exposure evaluation value from the long accumulation image to obtain a first long accumulation evaluation value; computing an exposure evaluation value from the long accumulation image according to a different method from that used to obtain the first long accumulation evaluation value to obtain a second long accumulation evaluation value; computing an exposure evaluation value from the short accumulation image to obtain a short accumulation evaluation value; obtaining a short accumulation target exposure time and short accumulation exposure control information from the short accumulation evaluation value and a first target level; obtaining a long accumulation evaluation value by synthesizing the first long accumulation evaluation value and the second long accumulation evaluation value in response to the short accumulation target exposure time using a predetermined method; acquiring long accumulation exposure control information from the long accumulation evaluation value and a second target level; and generating an image from the long accumulation image and the short accumulation image in response to the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 17 (depends on 16) The method according to claim 16, further comprising driving an image sensor from the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 18 (depends on 16) The method according to claim 16, wherein the long accumulation exposure control information and the short accumulation exposure control information contain information of exposure times. ### Claim 19 (depends on 16) The method according to claim 16, wherein the first target level and the second target level take the same value. ### Claim 20 (depends on 16) The method according to claim 16, wherein computing the exposure evaluation value to obtain the first long accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward a low-brightness side, and computing the exposure evaluation value to obtain the second long accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward a high-brightness side, and computing the exposure evaluation value to obtain the short accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward the high-brightness side, respectively. ### Claim 21 (depends on 20) The method according to claim 20, wherein the weights are equal in computing the second long accumulation evaluation value and the short accumulation evaluation value. ### Claim 22 (depends on 16) The method according to claim 16, wherein the long accumulation evaluation value is obtained by taking the first long accumulation evaluation value if the short accumulation target exposure time is less than a first threshold value, and taking the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than the first threshold value as the long accumulation evaluation value, respectively. ### Claim 23 (depends on 22) The method according to claim 22, wherein the first threshold value is a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 24 (depends on 22) The method according to claim 22, wherein, if an amount of PGA gain in the short accumulation image is larger, the first threshold value is smaller. ### Claim 25 (depends on 16) The method according to claim 16, wherein the long accumulation evaluation value is obtained by taking the first long accumulation evaluation value if the short accumulation target exposure time is less than a second threshold value, taking the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than a third threshold value larger than the second threshold value, and taking a value obtained by complementing the first long accumulation evaluation value and the second long accumulation evaluation value using a predetermined method if the short accumulation target exposure time is equal to or more than the second threshold value and less than the third threshold value as the long accumulation evaluation value, respectively. ### Claim 26 (depends on 25) The method according to claim 25, wherein the second threshold value takes a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 27 (depends on 25) The method according to claim 25, wherein, if an amount of PGA gain in the short accumulation image is larger, the second and third threshold values are smaller. ### Claim 28 (depends on 16) The method according to claim 16, wherein, if the short accumulation target exposure time is equal to or more than a fourth threshold value, in generating the image, not the composite image of the long accumulation image and the short accumulation image, but the long accumulation image itself is output to downstream processing. ### Claim 29 (depends on 28) The method according to claim 28, wherein the fourth threshold value is equal to or more than the first threshold value. ### Claim 30 (depends on 28) The method according to claim 28, wherein the fourth threshold value is equal to or more than the third threshold value. ### Claim 31 A non-transitory computer readable medium storing program code executable by a processing unit to perform operations comprising: imaging, within one field, at least a long accumulation image with a longer exposure time and a short accumulation image with a shorter exposure time; computing an exposure evaluation value from the long accumulation image to obtain a first long accumulation evaluation value; computing an exposure evaluation value from the long accumulation image according to a different method from that used to obtain the first long accumulation evaluation value to obtain a second long accumulation evaluation value; computing an exposure evaluation value from the short accumulation image to obtain a short accumulation evaluation value; obtaining a short accumulation target exposure time and short accumulation exposure control information from the short accumulation evaluation value and a first target level; obtaining a long accumulation evaluation value by synthesizing the first long accumulation evaluation value and the second long accumulation evaluation value in response to the short accumulation target exposure time using a predetermined method; acquiring long accumulation exposure control information from the long accumulation evaluation value and a second target level; and generating an image from the long accumulation image and the short accumulation image in response to the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 32 (depends on 31) The non-transitory computer readable medium according to claim 31, further comprising driving an image sensor from the long accumulation exposure control information and the short accumulation exposure control information. ### Claim 33 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein the long accumulation exposure control information and the short accumulation exposure control information contain information of exposure times. ### Claim 34 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein the first target level and the second target level take the same value. ### Claim 35 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein computing the exposure evaluation value to obtain the first long accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward a low-brightness side, and computing the exposure evaluation value to obtain the second long accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward a high-brightness side, and computing the exposure evaluation value to obtain the short accumulation evaluation value comprises computing the exposure evaluation value by putting a weight toward the high-brightness side, respectively. ### Claim 36 (depends on 35) The non-transitory computer readable medium according to claim 35, wherein the weights are equal in computing the second long accumulation evaluation value and the short accumulation evaluation value. ### Claim 37 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein the long accumulation evaluation value is obtained by taking the first long accumulation evaluation value if the short accumulation target exposure time is less than a first threshold value, and taking the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than the first threshold value as the long accumulation evaluation value, respectively. ### Claim 38 (depends on 37) The non-transitory computer readable medium according to claim 37, wherein the first threshold value is a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 39 (depends on 37) The non-transitory computer readable medium according to claim 37, wherein, if an amount of PGA gain in the short accumulation image is larger, the first threshold value is smaller. ### Claim 40 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein the long accumulation evaluation value is obtained by taking the first long accumulation evaluation value if the short accumulation target exposure time is less than a second threshold value, taking the second long accumulation evaluation value if the short accumulation target exposure time is equal to or more than a third threshold value larger than the second threshold value, and taking a value obtained by complementing the first long accumulation evaluation value and the second long accumulation evaluation value using a predetermined method if the short accumulation target exposure time is equal to or more than the second threshold value and less than the third threshold value as the long accumulation evaluation value, respectively. ### Claim 41 (depends on 40) The non-transitory computer readable medium according to claim 40, wherein the second threshold value takes a value exceeding the shortest exposure time of the long accumulation image and the longest exposure time of the short accumulation image. ### Claim 42 (depends on 40) The non-transitory computer readable medium according to claim 40, wherein, if an amount of PGA gain in the short accumulation image is larger, the second and third threshold values are smaller. ### Claim 43 (depends on 31) The non-transitory computer readable medium according to claim 31, wherein, if the short accumulation target exposure time is equal to or more than a fourth threshold value, in generating the image, not the composite image of the long accumulation image and the short accumulation image, but the long accumulation image itself is output to downstream processing. ### Claim 44 (depends on 43) The non-transitory computer readable medium according to claim 43, wherein the fourth threshold value is equal to or more than the first threshold value. ### Claim 45 (depends on 43) The non-transitory computer readable medium according to claim 43, wherein the fourth threshold value is equal to or more than the third threshold value.