# US 2023/0054440 A1 · System and Method for Hybrid LiDAR Segmentation with Outlier Detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054440 A1 - Document ID: US20230054440A1 - Status: Published application - Kind: A1 - Application number: 17/406,682 - Filed: 2021-08-19 - Published: 2023-02-23 - Earliest claimed: 2021-08-19 - Assignees: Argo AI, LLC - Applicants: Argo AI, LLC - Inventors: Satyen Rajpal; Jason Paul Ziglar - Web page: https://patentgrep.com/patent/US20230054440A1 ## CPC - G01S 17/89: for mapping or imaging - B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot - B60W 60/001: {Planning or execution of driving tasks} - G01S 7/4815: {using multiple transmitters} - G01S 17/931: of land vehicles - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/2193: {based on specific statistical tests} - G06F 18/23: Clustering techniques - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/20: Ensemble learning - G06T 7/11: Region-based segmentation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 2420/408: Radar; Laser, e.g. lidar - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Devices, methods, and systems may obtain at least one point cloud, segment points in the at least one point cloud into a plurality of segments, train a neural network using known segments and a first loss function to generate a first trained neural network, train the first trained neural network using outlier segments and a second loss function to generate a second trained neural network, and train an extended isolation forest by applying an extended isolation algorithm to features of the known segments and features of the outlier segments to generate an anomaly score for each segment. ## Claims ### Claim 1 A computer-implemented method, comprising: obtaining, with at least one processor, at least one point cloud; segmenting, with the at least one processor, points in the at least one point cloud into a plurality of segments; training, with the at least one processor, a neural network by: providing, as input to the neural network, the plurality of segments; receiving, as output from the neural network, for each segment, one or more first output features associated with one of (i) that segment being a known segment clustered in a known cluster of a plurality of known clusters and (ii) that segment being an outlier segment located in an outlier space located outside each known cluster of the plurality of known clusters; and modifying one or more parameters of the neural network according to a first loss function that depends on the one or more first output features associated with each known segment to generate a first trained neural network; training, with the at least one processor, the first trained neural network by: providing, as input to the first trained neural network, each outlier segment; receiving, as output from the first trained neural network, for each outlier segment, one or more second output features associated with that outlier segment; and modifying one or more parameters of the first trained neural network according to a second loss function different than the first loss function to generate a second trained neural network, wherein the second loss function depends on the one or more second output features associated with each outlier segment; and training, with the at least one processor, an extended isolation forest by applying an extended isolation algorithm to (i) the one or more second output features associated with each outlier segment and (ii) the one or more first output features associated with each known segment, wherein the extended isolation forest generates, for each segment of the plurality of segments, an anomaly score. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising using, by the at least one processor, a conditional random field (CRF) to segment the points in the at least one point cloud into the plurality of segments. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the neural network includes a multilayer perceptron (MLP) network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the second loss function is defined according to the following Equation: L outlier = 1 N o × C ⁢ ∑ o ∈ O ∑ c ∈ C [ δ o -  f o - u c  ] + 2 where No is a number of the outlier segments, C is a number of the plurality of clusters, δo is a hinge distance, fo is the one or more second output features associated the outlier segment, and uc is a mean embedding for a cluster c. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: providing, with the at least one processor, the second trained neural network and the extended isolation forest; obtaining, with the at least one processor, at least one further point cloud; segmenting, with the at least one processor, further points in the at least one further point cloud into one or more further segments; processing, with the at least one processor, using the second trained neural network, the one or more further segments to generate one or more further output features associated with the one or more further segments; and processing, with the at least one processor, using the extended isolation forest, the one or more further features to generate one or more further anomaly scores for the one or more further segments. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: controlling, with the at least one processor, based on the one or more further features and the one or more further anomaly scores for the one or more further segments, at least one autonomous driving operation of an autonomous vehicle. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the at least one point cloud is associated with a plurality of laser emitters of a LiDAR system. ### Claim 8 A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least one point cloud; segment points in the at least one point cloud into a plurality of segments; train a neural network by: providing, as input to the neural network, the plurality of segments; receiving, as output from the neural network, for each segment, one or more first output features associated with one of (i) that segment being a known segment clustered in a known cluster of a plurality of known clusters and (ii) that segment being an outlier segment located in an outlier space located outside each known cluster of the plurality of known clusters; and modifying one or more parameters of the neural network according to a first loss function that depends on the one or more first output features associated with each known segment to generate a first trained neural network; train the first trained neural network by: providing, as input to the first trained neural network, each outlier segment; receiving, as output from the first trained neural network, for each outlier segment, one or more second output features associated with that outlier segment; and modifying one or more parameters of the first trained neural network according to a second loss function different than the first loss function to generate a second trained neural network, wherein the second loss function depends on the one or more second output features associated with each outlier segment; and train an extended isolation forest by applying an extended isolation algorithm to (i) the one or more second output features associated with each outlier segment and (ii) the one or more first output features associated with each known segment, wherein the extended isolation forest generates, for each segment of the plurality of segments, an anomaly score. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein the at least one processor uses a conditional random field (CRF) to segment the points in the at least one point cloud into the plurality of segments. ### Claim 10 (depends on 8) The computer program product of claim 8, wherein the neural network includes a multilayer perceptron (MLP) network. ### Claim 11 (depends on 8) The computer program product of claim 8, wherein the second loss function is defined according to the following Equation: L outlier = 1 N o × C ⁢ ∑ o ∈ O ∑ c ∈ C [ δ o -  f o - u c  ] + 2 where No is a number of the outlier segments, C is a number of the plurality of clusters, δo is a hinge distance, fo is the one or more second output features associated the outlier segment, and uc is a mean embedding for a cluster c. ### Claim 12 (depends on 8) The computer program product of claim 8, wherein the at least one processor is further configured to: provide the second trained neural network and the extended isolation forest; obtain at least one further point cloud; segment further points in the at least one further point cloud into one or more further segments; process, using the second trained neural network, the one or more further segments to generate one or more further output features associated with the one or more further segments; and process, using the extended isolation forest, the one or more further features to generate one or more further anomaly scores for the one or more further segments. ### Claim 13 (depends on 12) The computer program product of claim 12, wherein the at least one processor is further configured to control, based on the one or more further features and the one or more further anomaly scores for the one or more further segments, at least one autonomous driving operation of an autonomous vehicle. ### Claim 14 (depends on 8) The computer program product of claim 8, wherein the at least one point cloud is associated with a plurality of laser emitters of a LiDAR system. ### Claim 15 A system, comprising: one or more sensors configured to capture one or more point clouds associated with an environment surrounding an autonomous vehicle; at least one processor configured to: store a segmentation neural network trained by: providing, as input to a neural network, a plurality of segments segmented from at least one point cloud; receiving, as output from the neural network, for each segment, one or more first output features associated with one of (i) that segment being a known segment clustered in a known cluster of a plurality of known clusters and (ii) that segment being an outlier segment located in an outlier cluster located outside each known cluster of the plurality of known clusters; and modifying one or more parameters of the neural network according to a first loss function that depends on the one or more first output features associated with each known segment to generate a first trained neural network; providing, as input to the first trained neural network, each outlier segment; receiving, as output from the first trained neural network, for each outlier segment, one or more second output features associated with that outlier segment; and modifying one or more parameters of the first trained neural network according to a second loss function different than the first loss function to generate the segmentation neural network, wherein the second loss function depends on the one or more second output features associated with each outlier segment; store an extended isolation forest trained by: applying an extended isolation algorithm to (i) the one or more second output features associated with each outlier segment and (ii) the one or more first output features associated with each known segment, wherein the extended isolation forest generates, for each segment of the plurality of segments, an anomaly score; segment further points in the one or more point clouds into one or more further segments; process, using the segmentation neural network, the one or more further segments to generate one or more further output features associated with the one or more further segments; and process, using the extended isolation forest, the one or more further features to generate one or more further anomaly scores for the one or more further segments. ### Claim 16 (depends on 15) The system of claim 15, wherein the at least one processor is further configured to use a conditional random field (CRF) to segment the further points in the one or more point clouds into the one or more further segments. ### Claim 17 (depends on 15) The system of claim 15, wherein the segmentation neural network includes a multilayer perceptron (MLP) network. ### Claim 18 (depends on 15) The system of claim 15, wherein the second loss function is defined according to the following Equation: L outlier = 1 N o × C ⁢ ∑ o ∈ O ∑ c ∈ C [ δ o -  f o - u c  ] + 2 where No is a number of the outlier segments, C is a number of the plurality of clusters, δo is a hinge distance, fo is the one or more second output features associated the outlier segment, and uc is a mean embedding for a cluster c. ### Claim 19 (depends on 15) The system of claim 15, wherein the at least one processor is further configured to control, based on the one or more further features and the one or more further anomaly scores for the one or more further segments, at least one autonomous driving operation of the autonomous vehicle. ### Claim 20 (depends on 15) The system of claim 15, wherein the one or more sensors include a LiDAR system.