# US 2010/0165136 A1 · Optical Imaging Systems And Methods Utilizing Nonlinear And/Or Spatially Varying Image Processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2010/0165136 A1 - Document ID: US20100165136A1 - Status: Published application - Kind: A1 - Application number: 12/294,664 - Filed: 2007-04-03 - Published: 2010-07-01 - Earliest claimed: 2006-04-03 - Assignees: OmniVision Technologies, Inc. - Inventors: Gregory E. Johnson; Edward R. Dowski, JR.; Kenneth S. Kubala; Ramakumar Narayanswamy; Hans Brandon Wach - Web page: https://patentgrep.com/patent/US20100165136A1 ## CPC - G02B 27/46: Systems using spatial filters - G06T 5/70: Denoising; Smoothing - G06T 5/73: Deblurring; Sharpening - G06T 7/136: involving thresholding - H04N 25/00: Circuitry of solid-state image sensors [SSIS]; Control thereof - H04N 25/683: by defect estimation performed on the scene signal, e.g. real time or on the fly detection - G06T 2207/20012: Locally adaptive - G06T 2207/20192: Edge enhancement; Edge preservation - H04N 25/67: applied to fixed-pattern noise, e.g. non-uniformity of response - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods include optics having one or more phase modifying elements that modify wavefront phase to introduce image attributes into an optical image. A detector converts the optical image to electronic data while maintaining the image attributes. A signal processor subdivides the electronic data into one or more data sets, classifies the data sets, and independently processes the data sets to form processed electronic data. The processing may optionally be nonlinear. Other imaging systems and methods include optics having one or more phase modifying elements that modify wavefront phase to form an optical image. A detector generates electronic data having one or more image attributes that are dependent on characteristics of the phase modifying elements and/or the detector. A signal processor subdivides the electronic data into one or more data sets, classifies the data sets and independently processes the data sets to form processed electronic data. ## Claims ### Claim 1 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to introduce one or more image attributes into an optical image; a detector that converts the optical image to electronic data while maintaining the image attributes; and a digital signal processor for subdividing the electronic data into one or more data sets, classifying the one or more data sets based at least on the image attributes, and independently processing the one or more data sets, based on results of the classifying, to form processed electronic data. ### Claim 2 (depends on 1) The imaging system of claim 1, wherein the imaging system has extended depth of field as compared to the system without the one or more phase modifying elements. ### Claim 3 (depends on 1) The imaging system of claim 1, wherein the digital signal processor is configured for classifying the one or more data sets based on (a) characteristics of the electronic data related to spatial regions of the optical image, and (b) the one or more image attributes. ### Claim 4 (depends on 3) The imaging system of claim 3, wherein the digital signal processor is further configured for classifying the one or more data sets based on power spectrum estimates for each of a plurality of identification subsets within the electronic data. ### Claim 5 (depends on 4) The imaging system of claim 4, wherein the digital signal processor is further configured for combining identification subsets with similar power spectrum estimates, to form the one or more data sets. ### Claim 6 (depends on 4) The imaging system of claim 4, wherein the digital signal processor is configured for (a) identifying dominant spatial frequencies in one or more of the power spectrum estimates, and (b) independently processing the one or more data sets by generating a filter for each of the data sets, based on the dominant spatial frequencies, to form the processed electronic data. ### Claim 7 (depends on 3) The imaging system of claim 3, wherein the digital signal processor is configured for identifying at least one edge in one of the spatial regions by utilizing one of Sobel, Prewitt, Roberts, Laplacian of Gaussian, zero cross and Canny methods. ### Claim 8 (depends on 1) The imaging system of claim 1, wherein the electronic data comprises one or more color channels. ### Claim 9 (depends on 8) The imaging system of claim 8, wherein the digital signal processor is configured for classifying the one or more data sets based on color information content of the one or more color channels. ### Claim 10 (depends on 8) The imaging system of claim 8, wherein the digital signal processor is further configured for classifying the one or more data sets based on spatial frequency information within the one or more color channels. ### Claim 11 (depends on 8) The imaging system of claim 8, wherein the digital signal processor is further configured for classifying the one or more data sets based on intensity information within the one or more color channels. ### Claim 12 (depends on 1) The imaging system of claim 1, wherein the electronic data comprises one or more intensity channels. ### Claim 13 (depends on 12) The imaging system of claim 12, wherein the digital signal processor is configured for classifying the one or more data sets based on intensity content of the one or more intensity channels. ### Claim 14 (depends on 1) The imaging system of claim 1, wherein the electronic data comprises a plurality of channels, each channel including at least color content and intensity content. ### Claim 15 (depends on 14) The imaging system of claim 14, wherein the digital signal processor is configured for classifying the one or more data sets based on the color content and intensity content of the plurality of channels. ### Claim 16 (depends on 1) The imaging system of claim 1, wherein the phase modifying element implements wavefront coding. ### Claim 17 (depends on 1) The imaging system of claim 1, wherein the one or more image attributes includes a blur that is removable by the digital signal processor. ### Claim 18 (depends on 1) The imaging system of claim 1, wherein the optics are characterized by a PSF having a shape that is substantially invariant with respect to misfocus. ### Claim 19 (depends on 1) The imaging system of claim 1, wherein the optics, the detector and the digital signal processor are located within a common system housing. ### Claim 20 (depends on 1) The imaging system of claim 1, wherein the optics and the detector are at a first location and the digital signal processor is at a second location. ### Claim 21 (depends on 1) The imaging system of claim 1, wherein the digital signal processor is further configured for operating on the electronic data in parallel with the detector converting the optical image to the electronic data. ### Claim 22 (depends on 1) The imaging system of claim 1, wherein the digital signal processor is configured for operating on the electronic data after the detector converts substantially all of the optical image to the electronic data. ### Claim 23 (depends on 1) The imaging system of claim 1, wherein the digital signal processor is further configured for classifying the one or more data sets in accordance with preset instructions. ### Claim 24 (depends on 23) The imaging system of claim 23, wherein the preset instructions are based on preferences of a user of the imaging system. ### Claim 25 (depends on 23) The imaging system of claim 23, wherein the preset instructions are based on the one or more image attributes. ### Claim 26 (depends on 1) The imaging system of claim 1, wherein the digital signal processor is additionally configured for recursively subdividing the electronic data into one or more data sets, classifying the one or more data sets, based at least on the image attributes, and independently processing the one or more data sets, based on results of the classifying, to form the processed electronic data. ### Claim 27 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to form an optical image; a detector that converts the optical image to electronic data having one or more image attributes that are dependent on characteristics of at least one of the phase modifying elements and the detector; and a digital signal processor for subdividing the electronic data into one or more data sets, classifying the one or more data sets, based at least in part on the one or more image attributes, and independently processing each of the one or more data sets, based on results of classifying, to form processed electronic data. ### Claim 28 (depends on 27) The imaging system of claim 27, wherein the one or more image attributes includes a blur induced by the one or more phase modifying elements. ### Claim 29 (depends on 27) The imaging system of claim 27, wherein the one or more image attributes includes one or more of shot noise and fixed pattern noise characterizing the detector. ### Claim 30 (depends on 27) The imaging system of claim 27, wherein the electronic data further includes a plurality of subspaces. ### Claim 31 (depends on 30) The imaging system of claim 30, wherein the plurality of subspaces includes one or more signal subspaces and noise subspaces. ### Claim 32 (depends on 27) The imaging system of claim 27, wherein the digital signal processor alters one or more of the image attributes to a degree that is independently adjustable for each of the data sets, based on the one or more image attributes of the data sets. ### Claim 33 (depends on 27) The imaging system of claim 27, wherein the digital signal processor alters one or more of the image attributes to a degree that is independently adjustable for each of the data sets, based on preset instructions. ### Claim 34 (depends on 33) The imaging system of claim 33, wherein the preset instructions are based on preferences of a user of the imaging system. ### Claim 35 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to introduce one or more image attributes into an optical image; a detector that converts the optical image to electronic data while maintaining the image attributes; and a digital signal processor configured for subdividing the electronic data into one or more data sets, classifying the one or more data sets based at least on the image attributes, and independently processing the one or more data sets, based on results of the classifying, to form processed electronic data, wherein the digital signal processor is further configured for altering one or more of the image attributes to a degree that is independently adjustable for each of the one or more data sets, based on preset instructions. ### Claim 36 (depends on 35) The imaging system of claim 35, wherein the preset instructions are based on one of preferences of a user of the imaging system and a characteristic of the one or more phase modifying elements. ### Claim 37 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to predeterministically affect an optical image; a detector that converts the optical image to electronic data; a digital signal processor for subdividing the electronic data into one or more data sets, classifying the data sets, based at least in part on a priori knowledge about how the phase modifying elements modify the wavefront phase, and independently processing each of the data sets, based on results of the classifying, to form processed electronic data. ### Claim 38 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to predeterministically affect an optical image; a detector that converts the optical image to electronic data; a digital signal processor for subdividing the electronic data into one or more data sets, independently processing each of the data sets, based on a priori knowledge about how the phase modifying elements modify the wavefront phase and on results of the classifying, to form processed electronic data. ### Claim 39 An imaging system, comprising optics, including one or more phase modifying elements, that alter wavefront phase and produce an optical image with at least one known image attribute; a detector that converts the optical image to electronic data that, while preserving the image attribute, is divisible into a plurality of data sets; and a digital signal processor that processes the data sets to modify the image attribute in a degree and manner that is independently adjustable for each one of the data sets, to generate processed electronic data. ### Claim 40 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to introduce one or more image attributes into an optical image; a detector that converts the optical image to electronic data while maintaining the image attributes; and a digital signal processor for determining one or more characteristics of the electronic data, and providing nonlinear processing of the electronic data to modify the image attribute and to form processed electronic data. ### Claim 41 (depends on 40) The imaging system of claim 40, wherein the imaging system has extended depth of field as compared to the system without the one or more phase modifying elements. ### Claim 42 (depends on 40) The imaging system of claim 40, wherein the digital signal processor utilizes a shock-filter routine for the nonlinear processing. ### Claim 43 (depends on 40) The imaging system of claim 40, wherein the digital signal processor implements one or more of a threshold operator, an edge enhancement operator, an inflection emphasis operator, a gradient operator, and a diffusion operator for the nonlinear processing. ### Claim 44 (depends on 40) The imaging system of claim 40, wherein the digital signal processor (a) implements two or more process operators, each process operator being one of a threshold operator, an edge enhancement operator, an inflection emphasis operator, a gradient operator, and a diffusion operator, (b) assigns a weight to each process operator and (c) sums the process operators according to the weight of each process operator. ### Claim 45 (depends on 44) The imaging system of claim 44, wherein the weight assigned to each process operator is optimized for one of spatial noise reduction and color noise reduction. ### Claim 46 (depends on 40) The imaging system of claim 40, wherein the one or more phase modifying elements implement wavefront coding. ### Claim 47 (depends on 40) The imaging system of claim 40, wherein the one or more image attributes includes introducing a blur that is removable by processing of the electronic data. ### Claim 48 (depends on 40) The imaging system of claim 40, wherein the one or more image attributes includes causing a PSF associated with the optics to have a shape that is substantially invariant with respect to misfocus. ### Claim 49 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to introduce one or more image attributes into an optical image; a detector that converts the optical image to electronic data while maintaining the image attributes; and a digital signal processor for subdividing the electronic data into one or more data sets, classifying the one or more data sets, based at least on the image attributes, and independently and nonlinearly processing the one or more data sets to form processed electronic data. ### Claim 50 A method for generating processed electronic data, comprising modifying phase of a wavefront from an object to introduce one or more image attributes into an optical image formed by an imaging system, converting the optical image to electronic data while maintaining the image attributes, subdividing the electronic data into one or more data sets, classifying the one or more data sets based at least on the one or more image attributes, and independently processing the one or more data sets to form processed electronic data. ### Claim 51 (depends on 50) Method of claim 50, wherein modifying phase extends a depth of field of the imaging system. ### Claim 52 (depends on 50) Method of claim 50, wherein classifying the one or more data sets is based on (a) characteristics of the electronic data related to spatial regions of the optical image, and (b) the one or more image attributes. ### Claim 53 (depends on 52) Method of claim 52, wherein classifying the one or more data sets is based on power spectrum estimates for each of a plurality of identification subsets within the electronic data. ### Claim 54 (depends on 53) Method of claim 53, wherein classifying comprises combining identification subsets with similar power spectrum estimates to form the one or more data sets. ### Claim 55 (depends on 53) Method of claim 53, wherein classifying comprises identifying dominant spatial frequencies in one or more of the power spectrum estimates, and independently processing comprises generating a corresponding filter for each of the data sets, based on the dominant spatial frequencies of each data set and filtering each of the data sets with its corresponding filter to form the processed electronic data. ### Claim 56 (depends on 52) Method of claim 52, wherein classifying comprises identifying at least one edge in one of the spatial regions by utilizing one of Sobel, Prewitt, Roberts, Laplacian of Gaussian, zero cross and Canny methods. ### Claim 57 (depends on 50) Method of claim 50, wherein converting the optical image to electronic data comprises generating one or more color channels. ### Claim 58 (depends on 57) Method of claim 57, wherein classifying further comprises classifying based on color information content of the one or more color channels. ### Claim 59 (depends on 57) Method of claim 57, wherein classifying further comprises classifying the one or more data sets based on spatial frequency information within the one or more color channels. ### Claim 60 (depends on 57) Method of claim 57, wherein classifying further comprises classifying the one or more data sets based on intensity information within the one or more color channels. ### Claim 61 (depends on 50) Method of claim 50, wherein modifying phase comprises utilizing wavefront coding. ### Claim 62 (depends on 50) Method of claim 50, wherein modifying phase comprises introducing a blur that is removable by digital signal processing as one of the one or more image attributes. ### Claim 63 (depends on 50) Method of claim 50, wherein modifying phase comprises causing a PSF of the imaging system to have a shape that is substantially invariant with respect to misfocus. ### Claim 64 (depends on 50) Method of claim 50, wherein the subdividing, classifying and independently processing occur in parallel with the modifying and converting. ### Claim 65 (depends on 50) Method of claim 50, wherein the modifying and converting occur at one time and the subdividing, classifying and independently processing occur at a later time. ### Claim 66 (depends on 50) Method of claim 50, wherein classifying is performed in accordance with preset instructions. ### Claim 67 (depends on 66) Method of claim 66, further comprising generating the preset instructions according to preferences of a user of the imaging system. ### Claim 68 (depends on 50) Method of claim 50, wherein the steps of subdividing, classifying and independently processing are performed recursively. ### Claim 69 A software product comprising instructions stored on computer-readable media, wherein the instructions, when executed by a computer, perform steps for processing electronic data generated by (a) modifying phase of a wavefront from an object to introduce one or more image attributes into an optical image formed by an imaging system and (b) converting the optical image to electronic data while maintaining the image attributes, the instructions comprising: instructions for subdividing the electronic data into one or more data sets; instructions for classifying the one or more data sets based at least on the one or more image attributes; and instructions for independently processing the one or more data sets to form the processed electronic data. ### Claim 70 (depends on 69) Software product of claim 69, wherein the instructions for classifying include instructions for evaluating one of (a) characteristics of the electronic data related to spatial regions of the optical image and (b) the one or more image attributes. ### Claim 71 (depends on 69) Software product of claim 69, wherein the instructions for classifying include instructions for generating a power spectrum estimate for each of a plurality of identification subsets within the electronic data. ### Claim 72 (depends on 71) Software product of claim 71, wherein the instructions for classifying include instructions for combining identification subsets with similar power spectrum estimates to form the one or more data sets. ### Claim 73 (depends on 71) Software product of claim 71, wherein the instructions for classifying include instructions for identifying dominant spatial frequencies in one or more of the power spectrum estimates, and the instructions for independently processing include instructions for generating a corresponding filter for each of the data sets, based on the dominant spatial frequencies of each data set and instructions for filtering each of the data sets with its corresponding filter to form the processed electronic data. ### Claim 74 (depends on 69) Software product of claim 69, wherein the instructions for classifying include instructions for identifying at least one edge in one of the spatial regions by utilizing one of Sobel, Prewitt, Roberts, Laplacian of Gaussian, zero cross and Canny methods. ### Claim 75 (depends on 69) Software product of claim 69, wherein the instructions for classifying include instructions for utilizing one of (a) color information content, (b) spatial frequency information and (c) intensity information of one or more color channels of the electronic data. ### Claim 76 (depends on 69) Software product of claim 69, wherein the instructions for subdividing, classifying and independently processing, respectively, include instructions for subdividing, classifying and independently processing in accordance with user preferences. ### Claim 77 (depends on 69) Software product of claim 69, further comprising instructions for recursively subdividing the electronic data into one or more data sets, classifying the one or more data sets based at least on the one or more image attributes; and processing the one or more data sets to form processed electronic data. ### Claim 78 An imaging system, comprising optics having one or more phase modifying elements that modify wavefront phase to introduce one or more image attributes into an optical image; a detector that converts the optical image to electronic data while maintaining the image attributes; and a digital signal processor for subdividing the electronic data into one or more data sets, independently processing the one or more data sets in one or more subspaces to form processed electronic data. ### Claim 79 (depends on 78) The imaging system of claim 78, wherein the digital signal processor is further configured for classifying the one or more data sets, and wherein the independently processing comprises independently processing the one or more subspaces based on results of the classifying. ### Claim 80 A method for generating processed electronic data, comprising modifying phase of a wavefront from an object to introduce one or more image attributes into an optical image formed by an imaging system; converting the optical image to electronic data while maintaining the image attributes; subdividing the electronic data into one or more data sets; and independently processing the one or more data sets in one or more subspaces to form processed electronic data.