# US 2022/0276384 A1 · Time-of-Flight Sensor with Structured Light Illuminator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0276384 A1 - Document ID: US20220276384A1 - Status: Published application - Kind: A1 - Application number: 17/739,064 - Filed: 2022-05-06 - Published: 2022-09-01 - Earliest claimed: 2018-11-01 - Assignees: Waymo LLC - Applicants: Waymo LLC - Inventors: Caner Onal; David Schleuning; Brendan Hermalyn; Simon Verghese; Alexander McCauley; Brandyn White; Ury Zhilinsky - Web page: https://patentgrep.com/patent/US20220276384A1 ## CPC - G01S 17/89: for mapping or imaging - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 7/4817: {relating to scanning} - G01S 7/4865: Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487) - G01S 7/497: Means for monitoring or calibrating - G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) - G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) - G01S 17/32: using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated - G01S 17/46: Indirect determination of position data - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/931: of land vehicles - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure relates to systems and methods that provide information about a scene based on a time-of-flight (ToF) sensor and a structured light pattern. In an example embodiment, a sensor system could include at least one ToF sensor configured to receive light from a scene. The sensor system could also include at least one light source configured to emit a structured light pattern and a controller that carries out operations. The operations include causing the at least one light source to illuminate at least a portion of the scene with the structured light pattern and causing the at least one ToF sensor to provide information indicative of a depth map of the scene based on the structured light pattern. ## Claims ### Claim 1 A sensor system comprising: at least one time-of-flight (ToF) sensor configured to receive light from a scene; at least one light source configured to emit a structured light pattern; and a controller that carries out operations, the operations comprising: dynamically adjusting the structured light pattern based on one or more retroreflective regions in the scene; causing the at least one light source to illuminate at least a portion of the scene with the structured light pattern; and causing the at least one ToF sensor to provide time of flight information indicative of a depth map of the scene based on the structured light pattern. ### Claim 2 (depends on 1) The sensor system of claim 1, wherein dynamically adjusting the structured light pattern comprises lowering illumination levels for portions of the scene where the one or more retroreflector regions are present. ### Claim 3 (depends on 1) The sensor system of claim 1, wherein the at least one ToF sensor comprises a plurality of complementary metal-oxide semiconductor (CMOS) or charge-coupled device (CCD) photosensitive elements. ### Claim 4 (depends on 1) The sensor system of claim 1, wherein the structured light pattern comprises at least one of: a predetermined spatial distribution of light, a predetermined temporal distribution of light, or a predetermined spectral distribution of light. ### Claim 5 (depends on 1) The sensor system of claim 1, wherein the structured light pattern comprises at least one of: a predetermined light pulse repetition rate, a predetermined light pulse duration, a predetermined light pulse intensity, or a predetermined light pulse duty cycle. ### Claim 6 (depends on 1) The sensor system of claim 1, wherein the at least one light source comprises at least one of: a laser diode, a light-emitting diode, a plasma light source, a strobe light, a solid-state laser, or a fiber laser. ### Claim 7 (depends on 1) The sensor system of claim 1, wherein dynamically adjusting the structured light pattern comprises selecting a desired structured light pattern from among a plurality of possible structured light patterns, wherein causing the at least one light source to illuminate at least a portion of the scene with the structured light pattern comprises illuminating the portion of the scene according to the desired structured light pattern. ### Claim 8 (depends on 1) The sensor system of claim 1, further comprising an imaging sensor, wherein the imaging sensor comprises a plurality of photosensitive elements, wherein the plurality of photosensitive elements comprises at least one million photosensitive elements, wherein the operations further comprise causing the imaging sensor to provide information indicative of an image of the scene based on the structured light pattern. ### Claim 9 (depends on 8) The sensor system of claim 8, wherein the operations further comprise determining a high-resolution depth map of the scene based on the depth map of the scene and the image of the scene. ### Claim 10 (depends on 8) The sensor system of claim 8, wherein the at least one ToF sensor, the imaging sensor, and the at least one light source are coupled to a common substrate. ### Claim 11 (depends on 1) The sensor system of claim 1, wherein the operations further comprise determining at least one inference about the scene based on the depth map of the scene. ### Claim 12 (depends on 11) The sensor system of claim 11, wherein the at least one inference comprises information about objects in an environment of a vehicle or an operating context of the vehicle. ### Claim 13 (depends on 11) The sensor system of claim 11, wherein the controller comprises at least one deep neural network, wherein the determining the at least one inference is performed by the at least one deep neural network. ### Claim 14 A system comprising: a plurality of sensor systems configured to be coupled to a vehicle, wherein each sensor system comprises: at least one time-of-flight (ToF) sensor; at least one imaging sensor, wherein the at least one ToF sensor and the at least one imaging sensor are configured to receive light from a scene; at least one light source configured to emit a structured light pattern; and a controller that carries out operations, the operations comprising: dynamically adjusting the structured light pattern based on one or more retroreflective regions in the scene; causing the at least one light source to illuminate at least a portion of the scene with the structured light pattern; causing the at least one ToF sensor to provide time of flight information indicative of a depth map of the scene based on the structured light pattern; and causing the imaging sensor to provide information indicative of an image of the scene based on the structured light pattern. ### Claim 15 (depends on 14) The system of claim 14, wherein the operations further comprise determining a high-resolution depth map of the scene based on the depth map of the scene and the image of the scene. ### Claim 16 (depends on 14) The system of claim 14, wherein at least one of the sensor systems comprises at least one ToF sensor and at least one imaging sensor in a common housing. ### Claim 17 A method comprising: dynamically adjusting a structured light pattern based on one or more retroreflective regions in a scene; causing at least one light source to illuminate the scene with the structured light pattern; receiving, from a time-of-flight (ToF) sensor, time of flight information about the scene based on the structured light pattern; determining a depth map of the scene based on the received information; and determining at least one inference about the scene based on the depth map of the scene. ### Claim 18 (depends on 17) The method of claim 17, wherein the at least one inference comprises information about objects in an environment of a vehicle or an operating context of the vehicle. ### Claim 19 (depends on 17) The method of claim 17, wherein dynamically adjusting the structured light pattern comprises selecting a desired structured light pattern from among a plurality of possible structured light patterns, wherein causing the at least one light source to illuminate the scene with the structured light pattern comprises illuminating the scene according to the desired structured light pattern. ### Claim 20 (depends on 17) The method of claim 17, wherein dynamically adjusting the structured light pattern further comprises adjusting the structured light pattern based on an amount of ambient light or a time of day.