# US 12,118,780 B2 · Pointeff method for urban object classification with LiDAR point cloud data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,780 B2
- Document ID: US12118780B2
- Status: Granted
- Kind: B2
- Application number: 18/352,246
- Filed: 2023-07-14
- Published: 2024-01-18
- Granted: 2024-10-15
- Earliest claimed: 2023-07-14
- Assignees: Guilin University of Technology
- Applicants: Guilin University of Technology
- Inventors: Guoqing Zhou; Yue Jiang; Haoyu Wang
- Web page: https://patentgrep.com/patent/US12118780B2

## CPC

- G06V 10/82: using neural networks
- G06V 20/176: {Urban or other man-made structures}
- G06T 7/11: Region-based segmentation
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/806: {of extracted features}
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/30181: Earth observation
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to a PointEFF method for urban object classification with LiDAR point cloud data, and belongs to the field of LiDAR point cloud classification. The method comprises: point cloud data segmentation; End-to-end feature extraction layer construction; External feature fusion layer construction; and precision evaluation. The PointEFF method for urban object classification with LiDAR point cloud data fuses point cloud hand-crafted descriptors with End-to-end features obtained from a network at an up-sampling stage of a model by constructing an External Feature Fusion module, which improves a problem of local point cloud information loss caused by interpolation operation in the up-sampling process of domain feature pooling methods represented by PointNet and PointNet++, greatly improves classification precision of the model in complex ground features, especially in rough surface ground features, and is capable of being better applied to the classification of urban ground features with complex ground feature types.

## Claims

### Claim 1

A PointEFF method for urban object classification with LiDAR point cloud data, comprising the following steps: step 1: segmenting point cloud data; step 2: constructing a PointEFF End-to-end feature extraction layer; step 3: constructing a PointEFF External feature fusion layer; and step 4: inputting data to obtain a point cloud category label for precision evaluation, wherein:
the End-to-end feature extraction layer is constructed according to the following steps:
the End-to-end feature extraction layer comprising a network encoder and a network decoder; processing and abstracting, by the encoder, a group of points through an abstract set operation to recursively extract multi-scale features of a point cloud local region; gradually recovering, by the decoder, a spatial dimension through a feature propagation operation, fusing the features extracted in the coding process, and completing input and output of the same scale on the premise of reducing information loss; and transferring, by the encoder and the decoder, features of the same scale through two groups of jump link modules;
the network encoder comprising thrice abstract set operations, wherein the abstract set consists of a sampling layer, a grouping layer and a feature extraction layer; firstly, inputting N LiDAR points with three-dimensional attributes (x, y, z) into the network, and selecting a point N′ from the sampling layer by an iterative Farthest Point Sampling algorithm to define N′ centroids of the local region;
then, in the grouping layer, setting a radius r by a query ball algorithm, and searching adjacent k points in the centroid range r to construct a local region; after implementing the sampling layer and the grouping layer, sampling the LiDAR points into N′ central clusters, each central cluster contains k points and 36-dimensional attributes thereof, and outputting a group of set of points with a size of N′×k×36; finally, encoding the local regions into feature vectors through the feature extraction layer; inputting the set of points into an MLP network, and outputting N′×k×C, wherein C is a feature extracted by MLP, max-pooling each central cluster to select the largest feature in each central cluster as a regional feature, and outputting N′×C;
carrying out thrice abstract set operations until a global feature of 1×1024 is output;
the network decoder consisting of thrice feature propagation operations and two groups of jump link modules, gradually recovering the spatial dimension by using an up-sampling operation, fusing the features extracted during the encoding process, and completing input and output of the same scale on the premise of reducing information loss; in a feature propagation layer, in order to propagate learned features from a sampling point to an original point, interpolating an N″×C′ dimensional feature map obtained by the encoder firstly by using an Inverse Distance Weighted algorithm, calculating distances from each point to be interpolated to all the points, calculating weights, and interpolating the number of points from N″ to N′ to obtain an interpolated N′×C′ dimensional feature map; then linking, by the jump link modules, the C dimensional feature map obtained by the encoder at the same scale to obtain an N′×(C′ +C) dimensional feature map through dimensional feature map; and finally, obtaining an N′ ×C′″ the multi-layer perceptron; and
obtaining an N×128 dimensional feature map after three feature propagation operations;
the PointEFF External feature fusion layer is constructed according to the following steps:
the External feature fusion layer comprising extraction of hand-crafted descriptors and the External Feature Fusion module; and selecting a fast point feature histogram as an input of the External Feature Fusion module;
obtaining a normal of point cloud by plane fitting with least square method, and establishing a local coordinate system between two points according to obtained normal vectors:
μ
=
η
s
v
=
μ
×
(
p
t
-
p
s
)

p
t
-
p
s

2
ω
=
μ
×
v
a difference between point normal pairs being capable of being expressed by the angles α, ϕ and θ:
α
=
v
·
η
t
ϕ
=
μ
·
(
p
t
-
p
s
)
d
θ
=
arctan
⁡
(
ω
·
η
t
,
μ
·
η
t
)
d
=

p
t
-
p
s

2
quantizing the angles α, ϕ and θ to form a point feature histogram PFH;
representing features of the fast point feature histogram as:
FPFH
⁡
(
p
q
)
=
PFH
⁡
(
p
q
)
+
1
k
⁢
∑
i
=
1
k
⁢
1
ω
k
·
PFH
⁡
(
p
k
)
after obtaining a fast point cloud point feature histogram, transferring the N×128 dimensional feature map obtained from the End-to-end feature extraction layer and the N×33 dimensional feature histogram into the External Feature Fusion module as inputs; in the External Feature Fusion module, the technical feature obtained by the End-to-end feature extraction layer being:
x∈R128
the feature of the fast point feature histogram being:
y∈R33
a new feature histogram obtained by a concatenate operation being:
z−[x,y]∈R128+33
after the concatenate operation is completed, obtaining the N×128 dimensional feature map through the multi-layer perceptron, that is, each point having 128 dimensional features; and
finally, inputting the feature map, and obtaining a point cloud category label through one-dimensional convolution, thus completing the PointEFF method for urban object classification with LiDAR point cloud data.

### Claim 2 (depends on 1)

The PointEFF method for urban object classification with LiDAR point cloud data according to claim 1, wherein the point cloud data is segmented according to the following steps:
setting a set of points as R, wherein a first point is p1=(x1, y1, z1)∈R, and the rest points are pR-1=(xR-1,yR-1,zR-1)∈R, and a Euclidean distance from pR−1 to p1 is:
dR-1=√{square root over ((xR-1−x1)2+(yR-1−y1)2+(zR-1−z1)2)}
according to the Euclidean distance from each sample point to p1 and a number of points m in a segmentation region, dividing a field of p1 as {p1, p1 . . . , pm} and then calculating distances from the rest points in the set of points to a point pm+1 iteratively until all the points in the set of points are divided.