# US 2021/0243352 A1 · Auto exposure of spatially-multiplexed-exposure high-dynamic-range image sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0243352 A1 - Document ID: US20210243352A1 - Status: Published application - Kind: A1 - Application number: 17/274,161 - Filed: 2019-09-05 - Published: 2021-08-05 - Earliest claimed: 2018-09-07 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Jon S. MCELVAIN; Walter C. GISH; Jonathan Robert MILLER; Gregory John WARD; Robin ATKINS - Web page: https://patentgrep.com/patent/US20210243352A1 ## CPC - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/72: Combination of two or more compensation controls - H04N 23/73: by influencing the exposure time - H04N 23/75: by influencing optical camera components - H04N 25/134: {based on three different wavelength filter elements} - H04N 25/533: by using differing integration times for different sensor regions - H04N 25/535: by dynamic region selection - H04N 25/583: with different integration times - H04N 25/589: with different integration times, e.g. short and long exposures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An auto exposure method for a spatially-multiplexed-exposure (SME) high-dynamic-range (HDR) image sensor includes (a) retrieving raw image data from an exposure of the spatially-multiplexed-exposure high-dynamic-range image sensor, (b) preprocessing long-exposure pixel values and short-exposure pixel values of the raw image data to remove therefrom long-exposure pixel values and short-exposure pixel values failing to meet one or more quality requirements, (c) synthesizing, into an high-dynamic-range histogram, the long-exposure pixel values remaining after the step of preprocessing and the short-exposure pixel values remaining after the step of preprocessing, (d) deriving a goodness metric from the high-dynamic-range histogram, (e) adjusting at least one of the long exposure time and the short exposure time, based at least in part upon the goodness metric, and (f) outputting the at least one of the long exposure time and the short exposure time, as adjusted, to the spatially-multiplexed-exposure high-dynamic-range image sensor. ## Claims ### Claim 1 An auto exposure method for a spatially-multiplexed-exposure (SME) high-dynamic-range (HDR) image sensor, comprising: retrieving raw image data from an exposure of the spatially-multiplexed-exposure high-dynamic-range image sensor, the raw image data including long-exposure pixel values from pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor characterized by a long exposure time and short-exposure pixel values from pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor characterized by a short exposure time; preprocessing the long-exposure pixel values and the short-exposure pixel values to remove therefrom long-exposure pixel values and short-exposure pixel values failing to meet one or more image quality requirements; synthesizing, into an high-dynamic-range histogram, the long-exposure pixel values remaining after the step of preprocessing and the short-exposure pixel values remaining after the step of preprocessing; deriving a goodness metric from the high-dynamic-range histogram; adjusting at least one of the long exposure time and the short exposure time, based at least in part upon the goodness metric; and outputting the at least one of the long exposure time and the short exposure time, as adjusted, to the spatially-multiplexed-exposure high-dynamic-range image sensor. ### Claim 2 (depends on 1) The auto exposure method of claim 1, the goodness metric including a variance, across the high-dynamic-range histogram, of contribution from individual bins of the high-dynamic-range histogram to total entropy of the high-dynamic-range histogram. ### Claim 3 (depends on 2) The auto exposure method of claim 2, the step of deriving comprising computing the contribution Si from each individual bin i of the high-dynamic-range histogram as Si=−pi log2(pi), wherein pi is count in the individual bin i. ### Claim 4 (depends on 2) The auto exposure method of claim 2 further comprising, prior to the step of synthesizing, removing an effect of noise from the long-exposure pixel values and the short-exposure pixel values. ### Claim 5 (depends on 4) The auto exposure method as claimed in claim 4, wherein the step of removing the effect of noise comprises deconvoluting, according to a noise model, each of (a) a first histogram of the long-exposure pixel values and (b) a second histogram of the short-exposure pixel values. ### Claim 6 (depends on 5) The auto exposure method as claimed in claim 5, an expectation value of noise variance is modeled with a linear function of pixel value. ### Claim 7 The auto exposure method of claim, further comprising performing a plurality of iterations of the steps of retrieving, preprocessing, synthesizing, deriving, adjusting, and outputting, to determine an optimal set of the long exposure time and the short exposure time that minimizes the variance. ### Claim 8 (depends on 7) The auto exposure method of claim 7, further comprising, prior to the step of performing the plurality of iterations, determining an optimal long exposure time; and the step of performing the plurality of iterations comprising determining an optimal short exposure time, while maintaining the long exposure time at the optimal long exposure time. ### Claim 9 (depends on 8) The auto exposure method of claim 8, the step of determining the optimal long exposure time comprising changing the long exposure time to minimize a noise discrepancy between the long-exposure pixel values and the short-exposure pixel values at a transition point that defines transition between use of the long-exposure pixel values and use of the short-exposure pixel values in demosaicing of an image captured by the spatially-multiplexed-exposure high-dynamic-range image sensor. ### Claim 10 (depends on 9) The auto exposure method of claim 9, the step of changing the long exposure time comprising: setting the long exposure time to an initial value and the short exposure time to a minimum value; and carrying out at least one iteration of: (a) performing the steps of retrieving, preprocessing, and synthesizing to generate a preliminary high-dynamic-range histogram, (b) estimating a target value of the long exposure time from the preliminary high-dynamic-range histogram, (c) for a plurality of values of the long exposure time in a range about the target value, computing a respective plurality of parameters indicative of the noise discrepancy that would result from the long exposure time, (d) deducing, from the values of the parameter, a revised value of the long exposure time that corresponds to the transition point coinciding with the bin associated with a smallest one of the values of the noise discrepancy, (e) unless the revised value of the long exposure time is greater than or equal to the initial value of the long exposure time, updating the initial value of the long exposure time to the revised value of the long exposure time and proceeding to a subsequent iteration, and (f) if the revised value of the long exposure time is greater than or equal to the initial value of the long exposure time, outputting the revised value of the long exposure time as the optimal long exposure time. ### Claim 11 (depends on 1) The auto exposure method of claim 1, further comprising detecting a face in the raw image data; and the step of adjusting comprising adjusting the set of the long exposure time and the short exposure time partly based upon the goodness metric and partly to improve exposure of the face. ### Claim 12 (depends on 1) The auto exposure method of claim 1, comprising: in the step of retrieving, further retrieving, as part of the raw image data, middle-exposure pixel values from pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor characterized by a middle exposure time that is shorter than the long exposure time and longer than the short exposure time; in the step of preprocessing, further preprocessing the middle-exposure pixel values to remove therefrom middle-exposure pixel values failing to meet one or more second quality requirements; in the step of synthesizing, further synthesizing into the high-dynamic-range histogram the middle-exposure pixel values remaining after the step of preprocessing; in the step of adjusting, further adjusting the middle exposure time; and in the step of outputting, further outputting the middle exposure time as adjusted. ### Claim 13 (depends on 1) The auto exposure method of claim 1, the spatially-multiplexed-exposure high-dynamic-range image sensor being a color image sensor having a plurality of color-specific pixel sets, each of the color-specific pixel sets being specifically sensitive to a respective one of a plurality of colors, each of the color-specific pixel sets including a subset of the pixels characterized by the long exposure time and a subset of the pixels characterized by the short exposure time, the auto exposure method further comprising: performing the steps of retrieving, preprocessing, synthesizing, and deriving for each of the color-specific pixel sets separately; and collectively evaluating each goodness metric determined for the color-specific pixel sets to generate one adjusted set of the long exposure time and the short exposure time applicable to all of the colors. ### Claim 14 (depends on 1) The auto exposure method of claim 1, the spatially-multiplexed-exposure high-dynamic-range image sensor being a color image sensor having a plurality of color-specific pixel sets, wherein each of the color-specific pixel sets is specifically sensitive to a respective one of a plurality of colors, and wherein each of the color-specific pixel sets includes a subset of the pixels characterized by the long exposure time and a subset of the pixels characterized by the short exposure time, wherein the method further comprises: in the step of retrieving, retrieving, for each of the color-specific pixel sets, respective color-specific long-exposure pixel values and respective color-specific short-exposure pixel values, and prior to the step of preprocessing, combining spatially adjacent ones of the color-specific long-exposure pixel values associated with different colors to generate the long-exposure pixel values, and combining spatially adjacent ones of the color-specific short-exposure pixel values associated with different colors to generate the short-exposure pixel values, wherein the long-exposure pixel values and the short-exposure pixel values are representative of luminance. ### Claim 15 (depends on 1) The auto exposure method of claim 1, the step of preprocessing comprising filtering, separately from each other, the long-exposure pixel values and the short-exposure pixel values. ### Claim 16 (depends on 15) The auto exposure method of claim 15, wherein the step of filtering comprises removing (a) from the long-exposure pixel values, long-exposure pixel values above a saturation-related threshold and (b) from the short-exposure pixel values, short-exposure pixel values below a noise-related threshold. ### Claim 17 (depends on 15) The auto exposure method of claim 15, wherein the step of preprocessing comprises comparing the long-exposure pixel values and the short-exposure pixel values, remaining after the step of filtering, to evaluate motion discrepancies and remove short-exposure pixel values and long-exposure values from spatial regions of the spatially-multiplexed-exposure high-dynamic-range image sensor affected by motion discrepancies in excess of a threshold motion discrepancy level. ### Claim 18 (depends on 17) The auto exposure method of claim 17, wherein the step of removing the effect of noise is applied, separately, to the long-exposure pixel values and short-exposure pixel values remaining after the step of removal the short-exposure pixel values and the long-exposure values from the spatial regions of the spatially-multiplexed-exposure high-dynamic-range image sensor affected by motion discrepancies in excess of a threshold motion discrepancy level. ### Claim 19 (depends on 1) The auto exposure method of claim 1, the step of synthesizing comprising: scaling to a common scale (a) a first histogram of the long-exposure pixel values remaining after the step of preprocessing, and (b) a second histogram of the short-exposure pixel values remaining after the step of preprocessing; removing from the second histogram, bins of short-exposure pixel values that, on the common scale, have center value less than or equal to center value of top-most bin of the first histogram; and after the step of removing, merging the first histogram and the second histogram to form the high-dynamic-range histogram. ### Claim 20 (depends on 1) The auto exposure method of claim 1, the step of preprocessing comprises scaling the long-exposure pixel values and the short-exposure pixel values to compensate for presence of an optical gradient filter between the spatially-multiplexed-exposure high-dynamic-range image sensor and a scene represented by the raw image data. ### Claim 21 (depends on 1) The auto exposure method of claim 1, comprising performing a plurality of iterations of the steps of retrieving, preprocessing, synthesizing, deriving, adjusting, and outputting, to determine an optimal set of the long exposure time and the short exposure time that optimizes the goodness metric. ### Claim 22 (depends on 21) The auto exposure method of claim 21, wherein each subsequent iteration is based upon raw image data acquired by the spatially-multiplexed-exposure high-dynamic-range image sensor using the adjusted set of the long exposure time and the short exposure time outputted to the spatially-multiplexed-exposure high-dynamic-range image sensor in a most recent prior iteration. ### Claim 23 (depends on 21) The auto exposure method of claim 21, further comprising executing the step of performing separately for a plurality of local regions of the spatially-multiplexed-exposure high-dynamic-range image sensor, to locally optimize a spatially varying set of the long exposure time and the short exposure time. ### Claim 24 (depends on 21) The auto exposure method of any of the claim 21, further comprising (1) in the step of retrieving, retrieving the raw image data from a frame of a raw video stream recorded by the spatially-multiplexed-exposure high-dynamic-range image sensor, (2) executing the steps of retrieving, preprocessing, synthesizing, deriving, adjusting, and outputting for the raw image data of a plurality of frames of the raw video stream to adjust, during capture of the raw video stream by the spatially-multiplexed-exposure high-dynamic-range image sensor, the optimal long set of the exposure time and the short exposure time, (3) for each frame of the raw video stream, producing an spatially-multiplexed-exposure high-dynamic-range image from the long-exposure pixel values and the short-exposure pixel values, so as to produce an spatially-multiplexed-exposure high-dynamic-range video stream from the raw video stream, and (4) post-processing the spatially-multiplexed-exposure high-dynamic-range video stream to reduce temporal brightness instability of the spatially-multiplexed-exposure high-dynamic-range video stream caused by adjustment of the optimal set of the long exposure time and the short exposure time during the capture of the video series. ### Claim 25 (depends on 24) The auto exposure method of claim 24, the step of post-processing comprising multiplying each high-dynamic-range image by ratio of (a) a fixed reference sensitivity to (b) sensitivity of the spatially-multiplexed-exposure high-dynamic-range image sensor at the long exposure time and the short exposure time used during capture of the frame associated with the high-dynamic-range image. ### Claim 26 (depends on 21) The auto exposure method of claim 21, further comprising (1) obtaining a raw video stream of frames recorded by the spatially-multiplexed-exposure high-dynamic-range image sensor, (2) analyzing temporal brightness variation of the raw video stream to predict a brightness of a later frame to be captured by the spatially-multiplexed-exposure high-dynamic-range image sensor, and (3) performing the iterations on at least some of the frames of the raw video stream, wherein each of the iterations further includes manipulating, based upon the brightness predicted in the step of analyzing, the optimal set of the long exposure time and the short exposure time to be optimal for the later frame. ### Claim 27 (depends on 1) A product for auto exposure of a spatially-multiplexed-exposure high-dynamic-range image sensor, comprising machine-readable instructions encoded in non-transitory memory, the machine-readable instructions including: data input instructions that, when executed by a processor, cause the processor to perform any of the methods of claim 1. ### Claim 28 (depends on 27) A spatially-multiplexed-exposure high-dynamic-range imaging system with auto exposure functionality, comprising a spatially-multiplexed-exposure high-dynamic-range image sensor, a processor, and the product of claim 27 for adjusting the long exposure time and the short exposure time of the spatially-multiplexed-exposure high-dynamic-range image sensor when the processor executes the machine-readable instructions of the product, wherein the machine-readable instructions of the product further include iteration instructions that, when executed by the processor, perform a plurality of iterations of executing the data input instructions, the preprocessing instructions, the synthesizing instructions, the metric instructions, the adjustment instructions, and the data output instructions, to determine and output to the spatially-multiplexed-exposure high-dynamic-range image sensor an optimal set of the long exposure time and the short exposure time that optimizes the goodness metric. ### Claim 29 An auto exposure method for a spatially-multiplexed-exposure high-dynamic-range image sensor comprising: (I) retrieving raw image data from an exposure of the spatially-multiplexed-exposure high-dynamic-range image sensor, wherein the raw image data includes long-exposure pixel values from long-exposure pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor characterized by a long exposure time and short-exposure pixel values from short-exposure pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor characterized by a short exposure time, (II) for each of a plurality of spatial regions of the spatially-multiplexed-exposure high-dynamic-range image sensor, evaluating saturation level of a selected long-exposure pixel in the spatial region, (III) for each of the spatial regions characterized by the long-exposure pixel value of the selected long-exposure pixel exceeding a saturation threshold, determining (i) a reduced short exposure time for the short-exposure pixels in the spatial region and (ii) an increased digital gain value that compensates for the reduced short exposure time, and (IV) for each of the spatial regions characterized by the long-exposure pixel value of the selected long-exposure pixel exceeding the saturation threshold, outputting the reduced short exposure time and the increased digital gain value to each short-exposure pixel in the spatial region. ### Claim 30 (depends on 29) The auto exposure method of claim 29, wherein each of the spatial regions comprises one long-exposure pixel and one associated short-exposure pixel for each spectral-sensitivity class of pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor. ### Claim 31 (depends on 29) The auto exposure method of claim 29, wherein each of the spatial regions comprises a plurality of the long-exposure pixels and a plurality of the short-exposure pixels for each spectral-sensitivity class of pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor. ### Claim 32 (depends on 29) The auto exposure method of claim 29, wherein the spatial regions collectively comprise all long-exposure pixels and all short-exposure pixels of the spatially-multiplexed-exposure high-dynamic-range image sensor. ### Claim 33 (depends on 28) A high-dynamic-range imaging system with auto exposure functionality comprising (I) a spatially-multiplexed-exposure high-dynamic-range image sensor with circuitry configured to separately adjust at least one exposure time for each of a plurality of spatial regions, (II) a processor, and (III) machine-readable instructions encoded in non-transitory memory, wherein the machine-readable instructions include (a) data input instructions that, when executed by the processor, cause the processor to perform any of the methods of claim 28.