# US 12,111,398 B2 · LiDAR and ambience signal fusion in lidar receiver > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,398 B2 - Document ID: US12111398B2 - Status: Granted - Kind: B2 - Application number: 17/693,713 - Filed: 2022-03-14 - Published: 2023-06-08 - Granted: 2024-10-08 - Earliest claimed: 2021-12-07 - Assignees: Beijing Voyager Technology Co., Ltd. - Applicants: Beijing Voyager Technology Co., Ltd. - Inventors: Yue Lu; Youmin Wang - Web page: https://patentgrep.com/patent/US12111398B2 ## CPC - G01S 17/89: for mapping or imaging - G01S 7/4816: {of receivers alone} - G01S 7/4817: {relating to scanning} - G01S 7/4868: {Controlling received signal intensity or exposure of sensor} - G01S 7/499: using polarisation effects - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/931: of land vehicles - G02B 26/0833: {the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD (G02B26/0825 takes precedence; micromechanical devices in general B81B)} - G02B 26/101: {with both horizontal and vertical deflecting means, e.g. raster or XY scanners (colour television using laser beams scanning a display screen H04N9/3129)} - G02B 27/0988: {Diaphragms, spatial filters, masks for removing or filtering a part of the beam} - G02B 27/106: {for splitting or combining a plurality of identical beams or images, e.g. image replication} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Embodiments of the disclosure provide a receiver of an optical sensing system, and an optical sensing method. The receiver includes a micro shutter array configured to sequentially receive a series of laser beams returned from an environment at a plurality of time points. The micro shutter array sequentially opens a portion of the micro shutter array at a specified location at each time point, to allow a respective laser beam to pass through the micro shutter array at that time point and to reflect the ambient light by a remaining portion of the micro shutter array at that time point. The receiver further includes a photodetector configured to detect the laser beam that passes through the micro shutter array at each time point to obtain point cloud data and an image sensor configured to receive the ambient light reflected by the remaining portion of the micro shutter array to obtain image data. The receiver also includes a controller configured to fuse the point cloud data obtained from the photodetector with the image data obtained from the image sensor. ## Claims ### Claim 1 A receiver for optical sensing, comprising: a micro shutter array configured to sequentially receive a series of laser beams returned from an environment at a plurality of time points, wherein the micro shutter array sequentially opens a portion of the micro shutter array at a specified location at each time point, to allow a respective laser beam to pass through the micro shutter array at that time point and to reflect ambient light by a remaining portion of the micro shutter array at that time point; a photodetector configured to detect the laser beam that passes through the micro shutter array at each time point to obtain point cloud data; an image sensor configured to receive the ambient light reflected by the remaining portion of the micro shutter array to obtain image data; and a controller coupled to the photodetector and the image sensor, configured to fuse the point cloud data obtained from the photodetector with the image data obtained from the image sensor. ### Claim 2 (depends on 1) The receiver of claim 1, wherein the receiver further comprises a beam splitter configured to direct the reflected ambient light towards the image sensor of the receiver. ### Claim 3 (depends on 1) The receiver of claim 1, wherein to fuse the point cloud data with the image data, the controller is further configured to: integrate the point cloud data and the image data; and detect an object in the environment based on the integrated point cloud data and image data. ### Claim 4 (depends on 3) The receiver of claim 3, wherein the point cloud data comprises a plurality of data points corresponding to the plurality of time points, wherein the image data comprises a plurality of image frames corresponding to the plurality of time points, wherein to integrate the point cloud data and the image data, the controller is further configured to: determine a specific location of the portion of the micro shutter array that opens at a time point; determine a pixel in the image frame of the same time point, wherein the pixel locates at a location in the image frame corresponding to the specific location in the micro shutter array; and determine an intensity value of the pixel based on the data point of the point cloud data corresponding to the same time point. ### Claim 5 (depends on 3) The receiver of claim 3, wherein to integrate the point cloud data and the image data, the controller is further configured to augment the point cloud data with color or texture information extracted from the image data. ### Claim 6 (depends on 1) The receiver of claim 1, wherein to fuse the point cloud data with the image data, the controller is further configured to: perform a first perception on the point cloud data for detecting an object in the environment; perform a second perception on the image data for detecting the object; and fuse the first perception and the second perception to detect the object. ### Claim 7 (depends on 6) The receiver of claim 6, wherein the point cloud data is 3D and the image data is 2D, wherein to fuse the first perception and the second perception to detect the object, the controller is further configured to project data of the object detected by the first perception in the 3D point cloud data onto data of the object detected by the second perception in the 2D image data. ### Claim 8 (depends on 1) The receiver of claim 1, wherein to fuse the point cloud data with the image data, the controller is further configured to apply a neural learning network to at least one of the point cloud data or the image data. ### Claim 9 (depends on 1) The receiver of claim 1, wherein the micro shutter array comprises a plurality of micro shutter elements arranged in a two-dimensional array. ### Claim 10 An optical sensing method, comprising: receiving a series of laser beams at a plurality of time points and an ambient light returned from an environment by a micro shutter array disposed along a light path of the series of laser beams returned from the environment, wherein the micro shutter array sequentially opens a portion of the micro shutter array at a specified location at each time point, to allow one of the series of laser beams to pass through the micro shutter array at that time point and to reflect the ambient light by a remaining portion of the micro shutter array at that time point; receiving, by a photodetector of the optical sensing system, the series of laser beams passed through sequentially opened portions of the micro shutter array to obtain point cloud data; receiving, by an image sensor of the optical sensing system, the ambient light reflected by the micro shutter array to obtain image data; and fusing the point cloud data obtained from the photodetector with the image data obtained from the image sensor. ### Claim 11 (depends on 10) The optical sensing method of claim 10, wherein fusing the point cloud data with the image data further comprises: integrating the point cloud data and the image data; and detecting an object in the environment based on the integrated point cloud data and image data. ### Claim 12 (depends on 11) The optical sensing method of claim 11, wherein the point cloud data comprises a plurality of data points corresponding to the plurality of time points, wherein the image data comprises a plurality of image frames corresponding to the plurality of time points, wherein integrating the point cloud data and the image data further comprises: determining a specific location of the portion of the micro shutter array that opens at a time point; determining a pixel in the image frame of the same time point, wherein the pixel locates at a location in the image frame corresponding to the specific location in the micro shutter array; and determining an intensity value of the pixel based on the data point of the point cloud data corresponding to the same time point. ### Claim 13 (depends on 11) The optical sensing method of claim 11, wherein integrating the point cloud data and the image data further comprises augmenting the point cloud data with color or texture information extracted from the image data. ### Claim 14 (depends on 10) The optical sensing method of claim 10, wherein fusing the point cloud data with the image data further comprises: performing a first perception on the point cloud data for detecting an object in the environment; performing a second perception on the image data for detecting the object; and fusing the first perception and the second perception to detect the object. ### Claim 15 (depends on 14) The optical sensing method of claim 14, wherein the point cloud data is 3D and the image data is 2D, wherein fusing the first perception and the second perception to detect the object further comprises projecting data of the object detected by the first perception in the 3D point cloud data onto data of the object detected by the second perception in the 2D image data. ### Claim 16 (depends on 10) The optical sensing method of claim 10, wherein fusing the point cloud data with the image data further comprises applying a neural learning network to at least one of the point cloud data or the image data. ### Claim 17 A controller for optical sensing, comprising: a communication interface configured to receive point cloud data from a photodetector and image data from an image sensor, wherein the point cloud data is formed by a series of laser beams returned from an environment that pass through a micro shutter array at a plurality of time points and detected by the photodetector, wherein the image data is formed by ambient light reflected by the micro shutter array; and a processor configured to fuse the point cloud data with the image data to detect an object in the environment, wherein to fuse the point cloud data with the image data, the processor is further configured to project data of the object detected based on the point cloud data onto data of the object detected based on the image data. ### Claim 18 (depends on 17) The controller of claim 17, wherein the micro shutter array sequentially opens a portion of the micro shutter array at a specified location at each time point, to allow a respective laser beam to pass through the micro shutter array at that time point and to reflect the ambient light by a remaining portion of the micro shutter array at that time point. ### Claim 19 (depends on 17) The controller of claim 17, wherein to fuse the point cloud data with the image data, the processor is further configured to: integrate the point cloud data and the image data; and detect the object in the environment based on the integrated point cloud data and image data. ### Claim 20 (depends on 17) The controller of claim 17, wherein to fuse the point cloud data with the image data, the processor is further configured to: perform a first perception on the point cloud data for detecting the object in the environment; perform a second perception on the image data for detecting the object; and fuse the first perception and the second perception to detect the object.