# US 12,223,745 B1 · Pavement element annotation method for point cloud data with fusion of height, device for constructing and pre-annotation point cloud data with fusion of height, electronic device and computer-readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,745 B1 - Document ID: US12223745B1 - Status: Granted - Kind: B1 - Application number: 18/653,908 - Filed: 2024-05-02 - Granted: 2025-02-11 - Earliest claimed: 2024-05-02 - Applicants: Molar Intelligence (Hangzhou) CO., LTD. - Inventors: Qunshu Lin; Minghao Liu; Shigang Qi; Xinjun Wu; Yi Yang; Chao Zhang; Zijian Zhao; Qijun Shao - Web page: https://patentgrep.com/patent/US12223745B1 ## CPC - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30241: Trajectory - G06T 2210/56: Particle system, point based geometry or rendering - G06T 2219/004: Annotating, labelling - G06V 2201/07: Target detection - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention discloses a pavement element annotation method for point cloud data with fusion of height, which comprises the following steps: constructing all single-frame point clouds into a joint point cloud in a global coordinate system based on the pose of each frame of single-frame point cloud; removing dynamic objects in the joint point cloud to obtain a static joint point cloud; transforming the static joint point cloud into an overhead image; pre-annotating the pavement elements in an overhead image by using the pavement element pre-annotation model; modifying the pre-annotated result to obtain an overhead image annotation; based on a ball query algorithm and a ground point algorithm, establishing a transformational relation between the pixels of the overhead image and the points of the static joint point cloud; transforming the overhead image annotation into a static joint point cloud annotation based on the transformational relation. ## Claims ### Claim 1 A pavement element annotation method for point cloud data with fusion of height, comprising the following steps: constructing all single-frame point clouds into a joint point cloud in a global coordinate system based on a pose of each frame of single-frame point cloud; removing dynamic objects in the joint point cloud to obtain a static joint point cloud containing a plurality of points, each point having a respective set of coordinates which indicates a position of the point relative to an origin; transforming the static joint point cloud into a matrix based on the respective coordinates of the plurality of points, wherein the matrix has a width and a length, and saving the matrix as an image, thereby providing an overhead image; pre-annotating pavement elements in the overhead image by using a pavement element pre-annotation model; modifying a pre-annotated result to obtain an overhead image annotation; establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm; and transforming the overhead image annotation into a static joint point cloud annotation based on the transformational relation. ### Claim 2 (depends on 1) The pavement element annotation method for point cloud data with fusion of height according to claim 1, wherein the step of removing dynamic objects in the joint point cloud to obtain a static joint point cloud comprises: removing dynamic targets in the joint point cloud based on a 3D target detection method; or, removing dynamic objects in the joint point cloud based on a visibility method. ### Claim 3 (depends on 2) The pavement element annotation method for point cloud data with fusion of height according to claim 2, wherein, the 3D target detection method comprises: calling a 3D target detection model to detect dynamic targets based on the joint point cloud, and obtaining a plurality of different types of target frames; using a motion model and an object correlation algorithm to track a motion trajectory of an object, and estimating a motion state of the object to optimize a 3D target detection result; finally, removing the points in the target frames of the dynamic targets; wherein, the dynamic targets comprise vehicles, pedestrians, riders and other targets. ### Claim 4 (depends on 2) The pavement element annotation method for point cloud data with fusion of height according to claim 2, wherein the visibility method comprises: transforming the joint point cloud and each frame of single-frame point cloud into a depth map and a sub-depth map; calling the depth map and sub-depth map to judge and remove the dynamic targets in the joint point cloud and each frame of single-frame point cloud; and recovering ground points that are mistakenly removed in the process of removing the dynamic targets. ### Claim 5 (depends on 1) The pavement element annotation method for point cloud data with fusion of height according to claim 1, wherein the step of transforming the static joint point cloud into a matrix comprises: determining a length and a width of the matrix based on a minimum of an X-axis coordinate, a maximum of the X-axis coordinate, a minimum of a Y-axis coordinate and a maximum of the Y-axis coordinate in the static joint point cloud, that is, the boundaries of the static joint point cloud; and using intensity information of the static joint point cloud from a top view angle as image color information for image drawing, and assigning the image color information to the matrix. ### Claim 6 (depends on 1) The pavement element annotation method for point cloud data with fusion of height according to claim 1, wherein the step of pre-annotating pavement elements in the overhead image by using a pavement element pre-annotation model comprises: the pavement element pre-annotation model using a convolutional neural network to extract features of the overhead images and generating a 2D scene feature map, and then using an instance-instance transformation mode to predict pavement elements and generating point sequences; wherein, the pavement elements comprise single and double solid lines, dashed lines, backflow lines, variable lane lines, road boundary lines, crosswalks, parking areas and arrows. ### Claim 7 (depends on 1) The pavement element annotation method for point cloud data with fusion of height according to claim 1, wherein, the step of establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm comprises: using the ground point algorithm to determine ground points of the static joint point cloud; transforming the coordinates (xi, yi) of a pavement element in the overhead image into corresponding coordinates (xp, yp) in the global coordinate system; using the ball query algorithm to find a point (xg, yg) that is a ground point closest to (xp, yp) in the global coordinate system, so as to determine zg, that is, a height of the point of the static joint point cloud, and establishing the transformational relation between the pixels of the overhead image and the points of the static joint point cloud. ### Claim 8 A device for constructing and pre-annotation point cloud data with fusion of height, comprising memory and a processor configured for: constructing all single-frame point clouds into a joint point cloud in a global coordinate system based on a pose of each frame of single-frame point cloud; removing dynamic targets in the joint point cloud to obtain a static joint point cloud containing a plurality of points, each point having a respective set of coordinates which indicates a position of the point relative to an origin; transforming the static joint point cloud into a matrix based on the respective coordinates of the plurality of points, wherein the matrix has a width and a length, and saving the matrix as an image, thereby providing an overhead image; pre-annotating pavement elements in the overhead image by a pavement element pre-annotation model; modifying a pre-annotated result to obtain an overhead image annotation; module configured for establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm; and transforming the overhead image annotation into a static joint point cloud annotation based on the transformational relation. ### Claim 9 (depends on 8) The device for constructing and pre-annotation point cloud data with fusion of height according to claim 8, wherein removing dynamic objects in the joint point cloud to obtain a static joint point cloud comprises: removing dynamic targets in the joint point cloud based on a 3D target detection method; or, removing dynamic objects in the joint point cloud based on a visibility method. ### Claim 10 (depends on 9) The device for constructing and pre-annotation point cloud data with fusion of height according to claim 9, wherein the 3D target detection method comprises: calling a 3D target detection model to detect dynamic targets based on the joint point cloud, and obtaining a plurality of different types of target frames; using a motion model and an object correlation algorithm to track a motion trajectory of an object, and estimating a motion state of the object to optimize a 3D target detection result; finally, removing the points in the target frames of the dynamic targets; wherein, the dynamic targets comprise vehicles, pedestrians, riders and other targets; the visibility method comprises: transforming the joint point cloud and each frame of single-frame point cloud into a depth map and a sub-depth map; calling the depth map and sub-depth map to judge and remove the dynamic targets in the joint point cloud and each frame of single-frame point cloud; and recovering ground points that are mistakenly removed in the process of removing the dynamic targets. ### Claim 11 (depends on 8) The device for constructing and pre-annotation point cloud data with fusion of height according to claim 8, wherein transforming the static joint point cloud into a matrix comprises: determining a length and a width of the overhead image based on a minimum of an X-axis coordinate, a maximum of the X-axis coordinate, a minimum of a Y-axis coordinate and a maximum of the Y-axis coordinate in the static joint point cloud, that is, the boundaries of the static joint point cloud; and using intensity information of the static joint point cloud from a top view angle as image color information for image drawing, and assigning the image color information to the matrix. ### Claim 12 (depends on 8) The device for constructing and pre-annotation point cloud data with fusion of height according to claim 8, wherein pre-annotating pavement elements in the overhead image by using a pavement element pre-annotation model comprises: the pavement element pre-annotation model using a convolutional neural network to extract features of the overhead images and generating a 2D scene feature map, and then using an instance-instance transformation mode to predict pavement elements and generating point sequences; wherein, the pavement elements comprise single and double solid lines, dashed lines, backflow lines, variable lane lines, road boundary lines, crosswalks, parking areas and arrows. ### Claim 13 (depends on 8) The device for constructing and pre-annotation point cloud data with fusion of height according to claim 8, wherein establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm comprises: using the ground point algorithm to determine ground points of the static joint point cloud; transforming the coordinates (xi, yi) of a pavement element in the overhead image into corresponding coordinates (xp, yp) in the global coordinate system; using the ball query algorithm to find a point (xg, yg) that is a ground point closest to (xp, yp) in the global coordinate system, so as to determine zg, that is, a height of the point of the static joint point cloud, and establishing the transformational relation between the pixels of the overhead image and the points of the static joint point cloud. ### Claim 14 An electronic device comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the following method: constructing all single-frame point clouds into a joint point cloud in a global coordinate system based on a pose of each frame of single-frame point cloud; removing dynamic objects in the joint point cloud to obtain a static joint point cloud containing a plurality of points, each point having a respective set of coordinates which indicates a position of the point relative to an origin; transforming the static joint point cloud into a matrix based on the respective coordinates of the plurality of points, wherein the matrix has a width and a length, and saving the matrix as an image, thereby providing an overhead image; pre-annotating pavement elements in the overhead image by using a pavement element pre-annotation model; modifying a pre-annotated result to obtain an overhead image annotation; establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm; and transforming the overhead image annotation into a static joint point cloud annotation based on the transformational relation. ### Claim 15 (depends on 14) The electronic device according to claim 14, wherein the step of removing dynamic objects in the joint point cloud to obtain a static joint point cloud comprises: removing dynamic targets in the joint point cloud based on a 3D target detection method; or, removing dynamic objects in the joint point cloud based on a visibility method. ### Claim 16 (depends on 15) The electronic device according to claim 15, wherein the 3D target detection method comprises: calling a 3D target detection model to detect dynamic targets based on the joint point cloud, and obtaining a plurality of different types of target frames; using a motion model and an object correlation algorithm to track a motion trajectory of an object, and estimating a motion state of the object to optimize a 3D target detection result; finally, removing the points in the target frames of the dynamic targets; wherein, the dynamic targets comprise vehicles, pedestrians, riders and other targets. ### Claim 17 (depends on 15) The electronic device according to claim 15, wherein the visibility method comprises: transforming the joint point cloud and each frame of single-frame point cloud into a depth map and a sub-depth map; calling the depth map and sub-depth map to judge and remove the dynamic targets in the joint point cloud and each frame of single-frame point cloud; and recovering ground points that are mistakenly removed in the process of removing the dynamic targets. ### Claim 18 (depends on 14) The electronic device according to claim 14, wherein the step of transforming the static joint point cloud into a matrix comprises: determining a length and a width of the overhead image based on a minimum of an X-axis coordinate, a maximum of the X-axis coordinate, a minimum of a Y-axis coordinate and a maximum of the Y-axis coordinate in the static joint point cloud, that is, the boundaries of the static joint point cloud; and using intensity information of the static joint point cloud from a top view angle as image color information for image drawing, and assigning the image color information to the matrix. ### Claim 19 (depends on 14) The electronic device according to claim 14, wherein the step of pre-annotating pavement elements in the overhead image by using a pavement element pre-annotation model comprises: the pavement element pre-annotation model using a convolutional neural network to extract features of the overhead images and generating a 2D scene feature map, and then using an instance-instance transformation mode to predict pavement elements and generating point sequences; wherein, the pavement elements comprise single and double solid lines, dashed lines, backflow lines, variable lane lines, road boundary lines, crosswalks, parking areas and arrows. ### Claim 20 (depends on 14) The electronic device according to claim 14, wherein the step of establishing a transformational relation between pixels of the overhead image and points of the static joint point cloud based on a ball query algorithm and a ground point algorithm comprises: using the ground point algorithm to determine ground points of the static joint point cloud; transforming the coordinates (xi, yi) of a pavement element in the overhead image into corresponding coordinates (xp, yp) in the global coordinate system; using the ball query algorithm to find a point (xg, yg) that is a ground point closest to (xp, yp) in the global coordinate system, so as to determine zg, that is, a height of the point of the static joint point cloud, and establishing the transformational relation between the pixels of the overhead image and the points of the static joint point cloud.