# US 12,000,692 B2 · Three-dimensional imaging and sensing using a dynamic vision sensor and pattern projection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,000,692 B2 - Document ID: US12000692B2 - Status: Granted - Kind: B2 - Application number: 17/310,753 - Filed: 2020-02-21 - Published: 2022-04-21 - Granted: 2024-06-04 - Earliest claimed: 2019-02-22 - Assignees: Prophesee - Applicants: Prophesee - Inventors: Guillaume Chican; Manuele Brambilla; Xavier Lagorce - Web page: https://patentgrep.com/patent/US12000692B2 ## CPC - G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns} - G06F 18/23: Clustering techniques - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 20/64: Three-dimensional [3D] objects - G06T 2207/10016: Video; Image sequence - G06T 2207/10152: Varying illumination - G06V 2201/12: Acquisition of 3D measurements of objects - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure generally relates to systems and methods for three-dimensional image sensing. More specifically, and without limitation, this disclosure relates to systems and methods for detecting three-dimensional images, and using asynchronous image sensors for detecting the same. In one implementation, at least one processor determines a plurality of patterns associated with a plurality of electromagnetic pulses emitted by a projector onto a scene; receives, from an image sensor, one or more first signals based on reflections caused by the plurality of electromagnetic pulses; detects one or more first events corresponding to one or more first pixels of the image sensor based on the received signals; based on the one or more first events, initializes one or more first events more state machines; receives, from the image sensor, one or more second signals corresponding to the reflections; detects one or more second events corresponding to one or more second pixels of the image sensor based on the received signals; determines candidates for connecting the one or more state machines more second events to the one or more first events; and determines three-dimensional points for the one or more first pixels and the one or more second pixels based on the candidates and the one or more state machines. ## Claims ### Claim 1 A system for detecting three-dimensional images, comprising: a projector configured to project a plurality of patterns of electromagnetic pulses onto a scene, each pattern indexed to spatial coordinates of the projector; and an image sensor comprising a plurality of pixels and configured to detect reflections of the patterns from the scene caused by the projected plurality of patterns of electromagnetic pulses, and index the detected reflected patterns by spatial coordinates of the pixels detecting the reflected patterns, the system further comprising: at least one processor configured to: determine a plurality of patterns associated with the plurality of patterns of electromagnetic pulses, receive one or more first signals from the image sensor based on the detected reflected patterns, detect one or more first events corresponding to one or more first pixels of the image sensor based on the received first signals, based on the one or more first events, initialize one or more state machines configured for tracking reflected patterns that move across the pixels of the image sensor, receive one or more second signals from the image sensor based on the detected reflected patterns, detect one or more second events corresponding to one or more second pixels of the image sensor based on the received second signals, determine candidates for connecting the one or more second events to the one or more first events, and determine three-dimensional image points for the one or more first pixels and the one or more second pixels based on the candidates and the one or more state machines. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more first events correspond to a start of the plurality of patterns associated with the plurality of patterns of electromagnetic pulses. ### Claim 3 (depends on 2) The system of claim 2, wherein the one or more second events correspond to an end of the plurality of patterns associated with the plurality of patterns of electromagnetic pulses. ### Claim 4 (depends on 1) The system of claim 1, wherein the plurality of patterns comprise at least two different amplitudes separated in time; at least two different wavelengths separated in time; at least two different pulse lengths separated by a length of time; or a plurality of pulses separated by different lengths of time. ### Claim 5 (depends on 1) The system of claim 1, wherein the plurality of patterns comprise pulses having at least one of selected frequencies, phase shifts, or duty cycles used to encode symbols. ### Claim 6 (depends on 1) The system of claim 1, wherein the projector is further configured to project the plurality of patterns of electromagnetic pulses to a plurality of spatial locations in the scene. ### Claim 7 (depends on 6) The system of claim 6, wherein at least one of the spatial locations corresponds to at least one of the one or more first pixels, and at least one other of the spatial locations corresponds to at least one of the one or more second pixels. ### Claim 8 (depends on 1) The system of claim 1, wherein the projector is further configured to project the plurality of patterns of electromagnetic pulses at a plurality of different projection times. ### Claim 9 (depends on 8) The system of claim 8, wherein at least one of the projection times corresponds to at least one of the one or more first events, and at least one other of the projection times corresponds to at least one of the one or more second events. ### Claim 10 (depends on 1) The system of claim 1, wherein each pixel of the image sensor comprises: a detector electrically connected to at least one first photosensitive element and configured to generate a trigger signal when an analog signal proportional to brightness of light impinging on the at least one first photosensitive element matches a condition, and at least one second photosensitive element configured to output a signal that is a function of brightness of light impinging on the at least one second photosensitive element in response to the trigger signal. ### Claim 11 (depends on 10) The system of claim 10, wherein the at least one first photosensitive element comprises the at least one second photosensitive element. ### Claim 12 (depends on 11) The system of claim 11, wherein: the one or more first signals have positive polarity when the condition is an increasing condition and negative polarity when the condition is a decreasing condition; and the at least one processor is further configured to perform at least one of: decoding polarities of the one or more first signals to obtain the one or more first events, or discarding any of the one or more first signals separated by an amount of time larger than a threshold or associated with a bandwidth not within a predetermined range. ### Claim 13 (depends on 1) The system of claim 1, further comprising an optical filter configured to block any reflections associated with a wavelength not within a predetermined range. ### Claim 14 (depends on 1) The system of claim 1, wherein the plurality of patterns comprises: a set of unique symbols encoded in electromagnetic pulses; a set of quasi-unique symbols encoded in electromagnetic pulses; or a set of quasi-unique symbols encoded in electromagnetic pulses, wherein the symbols are unique within a geometrically defined space. ### Claim 15 (depends on 14) The system of claim 14, wherein the geometrically defined space comprises an epipolar line of the system. ### Claim 16 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to store finalized state machines comprising the one or more initialized state machines and the candidates. ### Claim 17 (depends on 16) The system of claim 16, wherein the at least one processor is further configured to use the stored state machines in determining candidates for subsequent events. ### Claim 18 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to use the plurality of patterns and the one or more state machines to determine candidates for connecting the one or more second events to the one or more first events. ### Claim 19 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to determine the three-dimensional image points based on spatial coordinates of the projector and first and second pixels of the image sensor detecting the reflected patterns. ### Claim 20 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to determine the three-dimensional image by performing a triangulation based on a location of the pixel of the image sensor that received the reflected pattern or a location of a final pixel of the image sensor that received the reflected pattern and the state machine to which the final pixel is linked. ### Claim 21 An imaging system, comprising: a plurality of pixels, each pixel comprising a first photosensitive element; and a system for detecting three-dimensional images, the system further comprising: a projector configured to project a plurality of patterns of electromagnetic pulses onto a scene, each pattern indexed to spatial coordinates of the projector; an image sensor comprising a plurality of pixels and configured to detect reflections of the patterns from the scene caused by the projected plurality of patterns of electromagnetic pulses, and index the detected reflected patterns by spatial coordinates of the pixels detecting the reflected patterns; and at least one processor configured to: determine a plurality of patterns associated with the plurality of patterns of electromagnetic pulses, receive one or more first signals from the image sensor based on the detected reflected patterns, detect one or more first events corresponding to one or more first pixels of the image sensor based on the received first signals, based on the one or more first events, initialize one or more state machines configured for tracking reflected patterns that move across the pixels of the image sensor, receive one or more second signals from the image sensor based on the detected reflected patterns, detect one or more second events corresponding to one or more second pixels of the image sensor based on the received second signals, determine candidates for connecting the one or more second events to the one or more first events, and determine three-dimensional image points for the one or more first pixels and the one or more second pixels based on the candidates and the one or more state machines. ### Claim 22 (depends on 21) The imaging system of claim 21, wherein the at least one processor is further configured to determine the three-dimensional image points based on spatial coordinates of the projector and first and second pixels of the image sensor detecting the reflected patterns. ### Claim 23 (depends on 21) The imaging system of claim 21, wherein the at least one processor is further configured to determine the three-dimensional image points by performing a triangulation based on a location of the pixel of the image sensor that received the reflected pattern or a location of a final pixel of the image sensor that received the reflected pattern and the state machine to which the final pixel is linked. ### Claim 24 A method for detecting three-dimensional images, comprising: determining a plurality of patterns corresponding to a plurality of patterns of electromagnetic pulses projected by a projector onto a scene; receiving, from an image sensor comprising a plurality of pixels, one or more first signals based on reflections of the patterns from the scene caused by the projected plurality of patterns of electromagnetic pulses; detecting one or more first events corresponding to one or more first pixels of the image sensor based on the received first signals; based on the one or more first events, initializing one or more state machines configured for tracking reflected patterns that move across the pixels of the image sensor; receiving, from the image sensor, one or more second signals based on the reflections; detecting one or more second events corresponding to one or more second pixels of the image sensor based on the received second signals; determining candidates for connecting the one or more second events to the one or more first events; and determining three-dimensional image points for the one or more first pixels and the one or more second pixels based on the candidates and the one or more state machines. ### Claim 25 (depends on 24) The method of claim 24, wherein the determining the three-dimensional image points further comprises determining the three dimensional image points based on spatial coordinates of the projector and first and second pixels of the image sensor detecting the reflected patterns. ### Claim 26 (depends on 24) The method of claim 24, wherein the determining the three-dimensional image points includes performing a triangulation based on a location of the pixel of the image sensor that received the reflected pattern or a location of a final pixel of the image sensor that received the reflected pattern and the state machine to which the final pixel is linked.