# US 2019/0320105 A1 · Image reconstruction from image sensor output > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0320105 A1 - Document ID: US20190320105A1 - Status: Published application - Kind: A1 - Application number: 16/386,215 - Filed: 2019-04-16 - Published: 2019-10-17 - Earliest claimed: 2018-04-17 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Huixuan TANG; Song CHEN; Xinqiao LIU - Web page: https://patentgrep.com/patent/US20190320105A1 ## CPC - G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns - H04N 23/71: Circuitry for evaluating the brightness variation - G02B 27/0172: {characterised by optical features} - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 23/56: provided with illuminating means - H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values} - H04N 25/131: including elements passing infrared wavelengths - H04N 25/133: including elements passing panchromatic light, e.g. filters passing white light - H04N 25/17: Colour separation based on photon absorption depth, e.g. full colour resolution obtained simultaneously at each pixel location - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G02B 6/34: utilising prism or grating {(G02B6/293 takes precedence)} - G02B 2027/0118: {comprising devices for improving the contrast of the display / brillance control visibility} - G02B 2027/0134: {of stereoscopic type} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - H04N 23/53: of electronic viewfinders, e.g. rotatable or detachable - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Methods and systems for reconstructing images from sensor data are provided. In one example, a method comprises: receiving input data generated by photodiodes each associated with a channel having a target wavelength range for photon-to-charge conversion; obtaining, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range; and generating, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel. ## Claims ### Claim 1 An apparatus, comprising: a plurality of sensing units, each sensing unit comprising one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion; and an image processor configured to: receive input data generated by the photodiodes of the plurality of sensing units responsive to incident light; obtain, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range; and generate, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel. ### Claim 2 (depends on 1) The apparatus of claim 1, further comprising a calibration module configured to: obtain data representing a raw spectral response curve of each photodiode of the photodiodes, the raw spectral response curve representing a spectral response of the each photodiode operating in the respective sensing unit to light of a wavelength range, the data representing the raw spectral response curves being combinable with the plurality of channel coefficients of the each channel to generate a target spectral response curve for the each channel; and determine, for the each channel, a plurality of channel separation coefficients that maximizes a first portion of the target spectral response curve within the target wavelength range of the each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of the each channel, wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the raw spectral response curve of each photodiode includes a first portion within the target wavelength of the channel associated with the each photodiode and a second portion out of the target wavelength out of the channel associated with the each photodiode; wherein the calibration module is configured to, for the each channel: determine, for each raw spectral response of the raw spectral responses, a first portion that is within the target wavelength range of the each channel and a second portion that is outside the target wavelength range; determine an in-band matrix based on correlations among the first portions of the raw spectral responses; determine an out-of-band matrix based on correlations among the second portions of the raw spectral responses; and determine the plurality of channel separation coefficients that increases a first combination of the in-band matrix and the plurality of channel separation coefficients with respect to a second combination of the out-of-band matrix and the plurality of channel separation coefficients. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the first combination comprises a matrix multiplication of a transpose of a third matrix comprising the plurality of channel separation coefficients, the in-band matrix, and the third matrix; and wherein the second combination comprises a matrix multiplication of the transpose of the out-of-band matrix, the second matrix, and the third matrix. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein the determination of the plurality of coefficients comprises solving a generalized-Eigenvalue problem of maximizing a ratio between the first combination and the second combination. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the calibration module is configured to: determine, for the each channel, the target spectral response curve based on combining the raw spectral response curves with the plurality of channel coefficients of the each channel; and determine, for the each channel, a plurality of channel crosstalk coefficients that indicate crosstalk components of the channel output data of the each channel contributed by the plurality of photodiodes; wherein the plurality of channel crosstalk coefficients of the each channel is determined based on minimizing a difference between the spectral response curve of the each channel and a combination of the plurality of channel crosstalk coefficients of the each channel and the target spectral response curves; and wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel. ### Claim 7 (depends on 2) The apparatus of claim 2, wherein the image processor is configured to: generate pre-processed input data for the each channel based on performing a demosaicing operation on the input data; and generate, for the each channel, the output data based on combining the pre-processed input data of the each channel with the plurality of channel separation coefficients of the each channel to reconstruct an image for the each channel. ### Claim 8 (depends on 6) The apparatus of claim 6, wherein the image processor is configured to: determine, based on the input data, a first image model associated with a visible light wavelength range and a second image model associated with an infra-red light wavelength range; and generate the output data for the each channel based on minimizing, under a pre-determined constraint, a first difference between the first image model and a first combination of a first part of the output data associated with the visible light wavelength range and the plurality of channel crosstalk coefficients of the each channel and based on minimizing, under the pre-determined constraint, a second difference between the second image model and a second combination of a second part of the output data associated with the infra-red light wavelength range and the plurality of channel crosstalk coefficients of the each channel. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the pre-determined constraint comprises a constraint on an image gradient. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the calibration module is configured to generate the channel crosstalk coefficients in Fourier domain; and wherein the image processor is configured to: generate the pre-processed input data in the Fourier domain; generate the image formation model in the Fourier domain; and generate the output data for the each channel based on minimizing a difference between the image formation model in the Fourier domain and a combination of the output data and the plurality of channel crosstalk coefficients of the each channel in the Fourier domain under the pre-determined constraint. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein each sensing unit of the plurality of sensing units comprises a first photodiode and a second photodiode forming a stack along a direction perpendicular to a light receiving surface of the each sensing unit. ### Claim 12 (depends on 1) The apparatus of claim 1, wherein each sensing unit of the plurality of sensing units comprises a single photodiode. ### Claim 13 A method, comprising: receiving input data generated by photodiodes of a plurality of sensing units responsive to incident light, wherein each sensing unit comprises one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion; obtaining, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range; and generating, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel. ### Claim 14 (depends on 13) The method of claim 13, further comprising: obtaining data representing a raw spectral response curve of each photodiode of the photodiodes, the raw spectral response curve representing a spectral response of the each photodiode operating in the respective sensing unit to light of a wavelength range, the data representing the raw spectral response curves being combinable with the plurality of channel coefficients of the each channel to generate a target spectral response curve for the each channel; and determining, for the each channel, a plurality of channel separation coefficients that maximizes a first portion of the target spectral response curve within the target wavelength range of the each channel with respect to a second portion of the target spectral response curve out of the target wavelength range of the each channel, wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel. ### Claim 15 (depends on 14) The method of claim 14, wherein the raw spectral response curve of each photodiode includes a first portion within the target wavelength of the channel associated with the each photodiode and a second portion out of the target wavelength out of the channel associated with the each photodiode; wherein the method further comprises, for the each channel: determining, for each raw spectral response of the raw spectral responses, a first portion that is within the target wavelength range of the each channel and a second portion that is outside the target wavelength range; determining an in-band matrix based on correlations among the first portions of the raw spectral responses; determining an out-of-band matrix based on correlations among the second portions of the raw spectral responses; and determining the plurality of channel separation coefficients that increases a first combination of the in-band matrix and the plurality of channel separation coefficients with respect to a second combination of the out-of-band matrix and the plurality of channel separation coefficients. ### Claim 16 (depends on 15) The method of claim 15, wherein the first combination comprises a matrix multiplication of a transpose of a third matrix comprising the plurality of channel separation coefficients, the in-band matrix, and the third matrix; and wherein the second combination comprises a matrix multiplication of the transpose of the out-of-band matrix, the second matrix, and the third matrix. ### Claim 17 (depends on 16) The method of claim 16, wherein determining the plurality of coefficients comprises solving a generalized-Eigenvalue problem of maximizing a ratio between the first combination and the second combination. ### Claim 18 (depends on 17) The method of claim 17, further comprising: determining, for the each channel, the target spectral response curve based on combining the raw spectral response curves with the plurality of channel coefficients of the each channel; and determining, for the each channel, a plurality of channel crosstalk coefficients that indicate crosstalk components of the channel output data of the each channel contributed by the plurality of photodiodes; wherein the plurality of channel crosstalk coefficients of the each channel is determined based on minimizing a difference between the spectral response curve of the each channel and a combination of the plurality of channel crosstalk coefficients of the each channel and the target spectral response curves; and wherein the plurality of channel separation coefficients for the each channel is part of the plurality of channel coefficients for the each channel. ### Claim 19 (depends on 18) The method of claim 18, further comprising: determining, based on the input data, a first image model associated with a visible light wavelength range and a second image model associated with an infra-red light wavelength range; and generating the output data for the each channel based on minimizing, under a pre-determined constraint, a first difference between the first image model and a first combination of a first part of the output data associated with the visible light wavelength range and the plurality of channel crosstalk coefficients of the each channel and based on minimizing, under the pre-determined constraint, a second difference between the second image model and a second combination of a second part of the output data associated with the infra-red light wavelength range and the plurality of channel crosstalk coefficients of the each channel. ### Claim 20 A non-transitory computer readable medium storing instructions that, when executed by a hardware processor, causes the hardware processor to perform: receiving input data generated by photodiodes of a plurality of sensing units responsive to incident light, wherein each sensing unit comprises one or more photodiodes, each of the one or more photodiodes being associated with a channel having a target wavelength range for photon-to-charge conversion; obtaining, for each channel, a plurality of channel coefficients, the plurality of channel coefficients being configured to, when combined with the input data to generate channel output data for the each channel, increase a main component of the channel output data contributed by a part of the incident light within the target wavelength range of the each channel with respect to a crosstalk component of the channel output data contributed by a part of the incident light out of the target wavelength range; and generating, for the each channel, the channel output data based on combining the input data with the plurality of channel coefficients to reconstruct an image for the each channel.