# US 2020/0401823 A1 · Lidar-based detection of traffic signs for navigation of autonomous vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401823 A1 - Document ID: US20200401823A1 - Status: Published application - Kind: A1 - Application number: 16/905,334 - Filed: 2020-06-18 - Published: 2020-12-24 - Earliest claimed: 2019-06-19 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Derek Thomas Miller; Yu Zhang; Lin Yang - Web page: https://patentgrep.com/patent/US20200401823A1 ## CPC - G06N 3/08: Learning methods - B60W 60/001: {Planning or execution of driving tasks} - G01C 21/3811: {Point data, e.g. Point of Interest [POI]} - G06F 18/23: Clustering techniques - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06V 10/82: using neural networks - G06V 20/582: {of traffic signs} - G06V 20/64: Three-dimensional [3D] objects - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2556/40: High definition maps - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract According to an aspect of an embodiment, operations may comprise receiving a point cloud representing a region. The operations may also comprise identifying a cluster of points in the point cloud having a higher intensity than points outside the cluster of points. The operations may also comprise determining a bounding box around the cluster of points. The operations may also comprise identifying a traffic sign within the bounding box. The operations may also comprise projecting the bounding box to coordinates of an image of the region captured by a camera. The operations may also comprise employing a deep learning model to classify a traffic sign type of the traffic sign in a portion of the image within the projected bounding box. The operations may also comprise storing information regarding the traffic sign and the traffic sign type in a high definition (HD) map of the region. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving a point cloud representing a region, the point cloud captured by a LIDAR sensor mounted on a vehicle; identifying a cluster of points in the point cloud having a higher intensity than points outside the cluster of points; determining a bounding box around the cluster of points; identifying a traffic sign within the bounding box; projecting the bounding box to coordinates of an image of the region captured by a camera; employing a deep learning model to classify a traffic sign type of the traffic sign in a portion of the image within the projected bounding box; and storing information regarding the traffic sign and the traffic sign type in a high definition (HD) map of the region. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases precision over a purely image-based sign detection method. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases recall over a purely image-based sign detection method. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the deep learning model is further employed to output more accurate coordinates for the bounding box in order to store the more accurate coordinates in the HD map of the region. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein: the computer-implemented method further comprises determining whether an area of the cluster of points is greater than a threshold area to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the area of the cluster of points is greater than the threshold area. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein: the computer-implemented method further comprises determining whether a plane fitting through the cluster of points is at a vertical angle compared to a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the plane fitting through the cluster of points is with a threshold value of being at a vertical angle compared to the ground plane. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein: the computer-implemented method further comprises determining a height of the cluster of points above a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points is at a height of at least a threshold value above the ground plane. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein: the computer-implemented method further comprises determining whether the cluster of points has an expected sign shape to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points has an expected shape. ### Claim 9 One or more non-transitory computer-readable storage media storing instructions that in response to being executed by one or more processors, cause a computer system to perform operations, the operations comprising: receiving a point cloud representing a region, the point cloud captured by a LIDAR sensor mounted on a vehicle; identifying a cluster of points in the point cloud having a higher intensity than points outside the cluster of points; determining a bounding box around the cluster of points; identifying a traffic sign within the bounding box; projecting the bounding box to coordinates of an image of the region captured by a camera; employing a deep learning model to classify a traffic sign type of the traffic sign in a portion of the image within the projected bounding box; and storing information regarding the traffic sign and the traffic sign type in a high definition (HD) map of the region. ### Claim 10 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases precision over a purely image-based sign detection method. ### Claim 11 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases recall over a purely image-based sign detection method. ### Claim 12 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein the deep learning model is further employed to output more accurate coordinates for the bounding box in order to store the more accurate coordinates in the HD map of the region. ### Claim 13 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein: the operations further comprise determining whether an area of the cluster of points is greater than a threshold area to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the area of the cluster of points is greater than the threshold area. ### Claim 14 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein: the operations further comprise determining whether a plane fitting through the cluster of points is at a vertical angle compared to a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the plane fitting through the cluster of points is with a threshold value of being at a vertical angle compared to the ground plane. ### Claim 15 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein: the operations further comprise determining a height of the cluster of points above a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points is at a height of at least a threshold value above the ground plane. ### Claim 16 (depends on 9) The one or more non-transitory computer-readable storage media of claim 9, wherein: the operations further comprise determining whether the cluster of points has an expected sign shape to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points has an expected shape. ### Claim 17 A computer system comprising: one or more processors; and one or more non-transitory computer readable media storing instructions that in response to being executed by the one or more processors, cause the computer system to perform operations, the operations comprising: receiving a point cloud representing a region, the point cloud captured by a LIDAR sensor mounted on a vehicle; identifying a cluster of points in the point cloud having a higher intensity than points outside the cluster of points; determining a bounding box around the cluster of points; identifying a traffic sign within the bounding box; projecting the bounding box to coordinates of an image of the region captured by a camera; employing a deep learning model to classify a traffic sign type of the traffic sign in a portion of the image within the projected bounding box; and storing information regarding the traffic sign and the traffic sign type in a high definition (HD) map of the region. ### Claim 18 (depends on 17) The computer system of claim 17, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases precision over a purely image-based sign detection method. ### Claim 19 (depends on 17) The computer system of claim 17, wherein the identifying of the traffic sign based on the higher intensity of the cluster of points in the point cloud captured by the LIDAR sensor increases recall over a purely image-based sign detection method. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the deep learning model is further employed to output more accurate coordinates for the bounding box in order to store the more accurate coordinates in the HD map of the region. ### Claim 21 (depends on 17) The computer system of claim 17, wherein: the operations further comprise determining whether an area of the cluster of points is greater than a threshold area to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the area of the cluster of points is greater than the threshold area. ### Claim 22 (depends on 17) The computer system of claim 17, wherein: the operations further comprise determining whether a plane fitting through the cluster of points is at a vertical angle compared to a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the plane fitting through the cluster of points is with a threshold value of being at a vertical angle compared to the ground plane. ### Claim 23 (depends on 17) The computer system of claim 17, wherein: the operations further comprise determining a height of the cluster of points above a ground plane to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points is at a height of at least a threshold value above the ground plane. ### Claim 24 (depends on 17) The computer system of claim 17, wherein: the operations further comprise determining whether the cluster of points has an expected sign shape to determine whether the cluster of points represents a traffic sign; and the projecting of the bounding box to the coordinates of the image is performed in response to determining that the cluster of points has an expected shape.