# US 10,873,738 B2 · Multi-frame range gating for lighting-invariant depth maps for in-motion applications and attenuating environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,738 B2 - Document ID: US10873738B2 - Status: Granted - Kind: B2 - Application number: 16/537,305 - Filed: 2019-08-09 - Published: 2020-01-30 - Granted: 2020-12-22 - Earliest claimed: 2016-03-03 - Assignees: 4D Intellectual Properties, LLC - Applicants: 4D Intellectual Properties, LLC - Inventors: James E. Retterath - Web page: https://patentgrep.com/patent/US10873738B2 ## CPC - H04N 23/671: {in combination with active ranging signals, e.g. using light or sound signals emitted toward objects} - G01S 7/4863: Detector arrays, e.g. charge-transfer gates - G01S 17/18: wherein range gates are used - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G01S 17/931: of land vehicles - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - H04N 13/257: Colour aspects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 13/296: Synchronisation thereof; Control thereof - H04N 2213/005: Aspects relating to the "3D+depth" image format - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Range gated imaging systems utilize short light pulses and synchronized detector cycles to image objects that are contained within a distance range. Single-image range gating allows depth estimation for static scenes, but this technique is not well-suited for in-motion applications or attenuating environments. In various embodiments, a multi-frame range gating system uses a series of range gate frames that are produced within a brief timing window, whereby each range-gate frame illuminates a different lighting-invariant depth range within the field of view. Multi-frame range gate cycles of as little as three frames are sufficient to produce precise lighting-invariant depth information for each pixel in a sensor for in-motion applications. Lighting-invariant depth information is produced with multi-frame range gate cycles of as little as four frames for in-motion applications for attenuating environments like fog, rain, dust, smoke, smog, and underwater. ## Claims ### Claim 1 An active image sensor system configured to generate a lighting-invariant depth map of a scene comprising: at least one emitter configured to emit a set of light pulses toward the scene; an array of detectors configured to receive light for a field of view that includes at least a portion of the scene; control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least three successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at least a second time, a third time and a fourth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time and the second time, between the first time and the third time and between the first time and the fourth time; and a processing system operably coupled to the control circuitry and the one or more frame buffers, the processing system configured to generate the lighting-invariant depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least three successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array: analyze the at least three successive frames of data for the detector and determine a minimum non-zero intensity value due to ambient light, a maximum intensity value, and a frame of the at least three successive frames at which the maximum intensity value occurs; and determine a distance to an object based on the frame at which the maximum intensity value occurs. ### Claim 2 (depends on 1) The active image sensor system of claim 1, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light. ### Claim 3 (depends on 1) The active image sensor system of claim 1, wherein the emitter/detector cycle is less than 50 microseconds. ### Claim 4 (depends on 1) The active image sensor system of claim 1, wherein the image sensor system is in motion relative to the scene. ### Claim 5 (depends on 1) The active image sensor system of claim 1, wherein the distance is determined based on a calculation of a time of flight. ### Claim 6 (depends on 1) The active image sensor system of claim 1, wherein the set of light pulses is a single light pulse. ### Claim 7 (depends on 1) The active image sensor system of claim 1, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp. ### Claim 8 (depends on 1) The system of claim 1, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of: ultraviolet (UV)—100-400 nm, visible—400-700 nm, near infrared (NIR)—700-1400 nm, infrared (IR)—1400-8000 nm, long-wavelength IR (LWIR)—8 um-15 um, or far IR (FIR)—15 um-1000 um. ### Claim 9 (depends on 1) The active image sensor system of claim 1, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements. ### Claim 10 (depends on 1) The active image sensor of claim 1, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels. ### Claim 11 An active image sensor system configured to generate a lighting-invariant depth map of a scene within an attenuating environment comprising: at least one emitter configured to emit a set of light pulses toward the scene; an array of detectors configured to receive light for a field of view that includes at least a portion of the scene, wherein the light includes a varying amount of light due to ambient light and to light pulse reflections from the attenuating environment; control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least five successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at a second time, a third time, a fourth time, a fifth time and a sixth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time and the second time, between the first time and the third time, between the first time and the fourth time, between the first time and the fifth time and between the first time and the sixth time to generate the at least five successive frames of data; and a processing system operably coupled to the control circuitry and the one or more frame buffers to generate the lighting-invariant depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least five successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array: analyze the at least five successive frames of data for the detector and determine a minimum intensity value due to ambient light, a maximum intensity value due to light reflected from the attenuating environment, and a frame of the at least five successive frames at which the minimum intensity value occurs; and determine a distance to an object based on the frame at which the minimum intensity value occurs. ### Claim 12 (depends on 11) The active image sensor system of claim 11, wherein the attenuating environment is selected from the set that includes one or more of fog, rain, dust, smoke, smog, and underwater. ### Claim 13 (depends on 11) The active image sensor system of claim 11, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light. ### Claim 14 (depends on 11) The active image sensor system of claim 11, wherein the emitter/detector cycle is less than 50 microseconds. ### Claim 15 (depends on 11) The active image sensor system of claim 11, wherein the image sensor system is in motion relative to the scene. ### Claim 16 (depends on 11) The active image sensor system of claim 11, wherein the distance is determined based on a calculation of a time of flight. ### Claim 17 (depends on 11) The active image sensor system of claim 11, wherein the set of light pulses is a single light pulse. ### Claim 18 (depends on 11) The active image sensor system of claim 11, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp. ### Claim 19 (depends on 11) The system of claim 11, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of: ultraviolet (UV)—100-400 nm, visible—400-700 nm, near infrared (NIR)—700-1400 nm, infrared (IR)—1400-8000 nm, long-wavelength IR (LWIR)—8 um-15 um, or far IR (FIR)—15 um-1000 um. ### Claim 20 (depends on 11) The active image sensor system of claim 11, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements. ### Claim 21 (depends on 11) The active image sensor of claim 11, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels. ### Claim 22 An active image sensor system configured to generate a depth map of a scene within an attenuating environment comprising: at least one emitter configured to emit a set of light pulses toward the scene; an array of detectors configured to receive light for a field of view that includes at least a portion of the scene, wherein the light includes a varying amount of light due to ambient light and to light pulse reflections from the attenuating environment; control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least four successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at a second time, a third time, a fourth time and a fifth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time second time, between the first time and the third time, between the first time and the fourth time, and between the first time and the fifth time to generate the at least four successive frames of data; and a processing system operably coupled to the control circuitry and the one or more frame buffers to generate the depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least five successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array: analyze the at least four successive of data for the detector and determine a minimum intensity value due to ambient light, a maximum intensity value due to light reflected from the attenuating environment, and a frame of the at least four successive frames, exclusive of a first frame of the at least four successive frames, at which the minimum intensity value occurs; and determine a distance to an object based on the frame at which the minimum intensity value occurs. ### Claim 23 (depends on 22) The active image sensor system of claim 22, wherein the attenuating environment is selected from the set that includes one or more of fog, rain, dust, smoke, smog, and underwater. ### Claim 24 (depends on 22) The active image sensor system of claim 22, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light. ### Claim 25 (depends on 22) The active image sensor system of claim 22, wherein the emitter/detector cycle is less than 50 microseconds. ### Claim 26 (depends on 22) The active image sensor system of claim 22, wherein the image sensor system is in motion relative to the scene. ### Claim 27 (depends on 22) The active image sensor system of claim 22, wherein the distance is determined based on a calculation of a time of flight. ### Claim 28 (depends on 22) The active image sensor system of claim 22, wherein the set of light pulses is a single light pulse. ### Claim 29 (depends on 22) The active image sensor system of claim 22, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp. ### Claim 30 (depends on 22) The system of claim 22, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of: ultraviolet (UV)—100-400 nm, visible—400-700 nm, near infrared (NIR)—700-1400 nm, infrared (IR)—1400-8000 nm, long-wavelength IR (LWIR)—8 um-15 um, or far IR (FIR)—15 um-1000 um. ### Claim 31 (depends on 22) The active image sensor system of claim 22, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements. ### Claim 32 (depends on 22) The active image sensor of claim 22, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels.