# US 11,113,835 B2 · Plant point cloud acquisition, registration and optimization method based on TOF camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,835 B2 - Document ID: US11113835B2 - Status: Granted - Kind: B2 - Application number: 16/885,682 - Filed: 2020-05-28 - Published: 2020-12-10 - Granted: 2021-09-07 - Earliest claimed: 2019-06-06 - Assignees: Zhejiang University - Applicants: Zhejiang University - Inventors: Haiyan Cen; Yong He; Zhihong Ma; Hui Fang - Web page: https://patentgrep.com/patent/US11113835B2 ## CPC - G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration - G06T 7/60: Analysis of geometric attributes - G06T 5/70: Denoising; Smoothing - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 20/188: {Vegetation} - G06V 20/64: Three-dimensional [3D] objects - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06V 2201/12: Acquisition of 3D measurements of objects - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present invention discloses a plant point cloud acquisition, registration, and optimization method based on a time of flight (TOF) camera, which includes the following steps: (1) placing a to-be-tested plant on a turntable, adjusting a view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant; (2) turning on the turntable so that it rotates automatically, and enabling the TOF camera to acquire point cloud data of the to-be-tested plant at intervals; (3) performing real-time preprocessing on each frame of point cloud data acquired by the TOF camera; (4) performing registration and optimization on every two adjacent frames of point cloud data, and then integrating the data to obtain complete plant point cloud data; and (5) using statistical filtering to remove the discrete noise in the plant point cloud data obtained in the registration and optimization process to obtain final point cloud data. ## Claims ### Claim 1 A plant point cloud acquisition, registration, and optimization method based on a time of flight (TOF) camera, comprising: (1) placing a to-be-tested plant on a turntable, adjusting a view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant; (2) turning on the turntable so that it rotates automatically, and enabling the TOF camera to acquire point cloud data of the to-be-tested plant at intervals; (3) performing real-time preprocessing on each frame of point cloud data acquired by the TOF camera; (4) performing registration and optimization on every two adjacent frames of point cloud data, and then integrating the data to obtain complete plant point cloud data, comprising: (4-1) transforming a coordinate system of a second point cloud P2 to a coordinate system of a first point cloud P1 to obtain a point cloud P2′; (4-2) triangulating P1 and P2′, removing boundary points that do not form a triangular patch, but retaining points inside that do not form a triangular patch; (4-3) searching for a triangular patch of P2′ within the neighborhood of each triangular patch in P1, and checking whether the triangular patch in P1 intersects with or is in parallel to the triangular patches within the neighborhood; (4-4) according to the intersection and parallelism relationship, adjust a point cloud set of P2′ that is within the neighborhood of the triangular patch in P1 to obtain a point cloud P2″; (4-5) adding points in P2′ that do not form a triangular patch to P2″, and then performing downsampling to obtain a point set P2′deal; (4-6) transforming a coordinate system of an i-th point cloud Pi to the coordinate system of the first point cloud P1 to obtain a point cloud Pi′, and repeating steps (4-2) to (4-5) to obtain Pi′deal, wherein i≥3; and (4-7) integrating P1, P2′deal, and Pi′deal to obtain the complete plant point cloud data; and (5) using statistical filtering to remove the discrete noise in the plant point cloud data obtained in the registration and optimization process to obtain final point cloud data. ### Claim 2 (depends on 1) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1, wherein step (1) comprises: (1-1) calibrating the TOF camera with Zhang Zhengyou method, placing the to-be-tested plant on the turntable, adjusting the view angle of the TOF camera, and aligning the TOF camera with the to-be-tested plant; and (1-2) using a plane fitting method to obtain a fitted plane of a tabletop, and obtaining an angle between a normal vector of the fitted plane of the tabletop and each axis of a right-hand coordinate system. ### Claim 3 (depends on 2) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 2, wherein step (1-2) comprises: (1-2a) performing extreme pass-through filtering on each frame of point cloud data so that only part of the visible tabletop is retained; (1-2b) obtaining multiple fitted planes through random sample consensus (RANSAC) plane fitting, obtaining a normal vector of each fitted plane, and determining the fitted plane of the tabletop based on an angle relationship between the normal vector of the fitted plane and the y-axis of the right-hand coordinate system and a point cloud quantity threshold, wherein the fitted plane of the tabletop meets the following formulas: min{θ,θ=arcos({right arrow over (n)}fitted plane,y)} (1) max{ny,ny∈{nyi,i=1,2 . . . m}} (2) numplane≥numthreshold (3) wherein {right arrow over (n)}fitted plane is the normal vector of the fitted plane; y is the y-axis of the right-hand coordinate system; θ is the angle between the normal vector of the fitted plane and the y-axis of the right-hand coordinate system; nyi is a y-axis component of the normal vector of each fitted plane, m is a quantity of fitted planes, and ny is a y-axis component of a normal of the fitted plane; numplane is a quantity of point clouds of the fitted plane; and numthreshold is the point cloud quantity threshold; and (1-2c) after determining the fitted plane of the tabletop, obtaining the angle between the normal vector of the fitted plane of the tabletop and each axis of the right-hand coordinate system. ### Claim 4 (depends on 1) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1, wherein step (3) comprises: (3-1) rotating each frame of the point cloud data acquired by the TOF camera according to an angle between a normal vector of a fitted plane of a tabletop and each axis of a right-hand coordinate system; (3-2) performing pass-through filtering on the rotated point cloud data to remove background and obtain an unordered point cloud; and (3-3) using bilateral filtering to smoothen the unordered point cloud. ### Claim 5 (depends on 1) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1, wherein the step (4-2) of searching for boundary points based on a KD tree and normal vector method comprises: (4-2a) searching for neighboring points of a point that does not form a triangular patch through the KD tree, wherein when a quantity of neighboring points of a point is less than a threshold, the point is an outlier; and (4-2b) according to the quantity of neighboring points of the point that does not form a triangular patch, using a PCA method to calculate a normal vector of the point, determining a connection line between a view point and the point, and calculating an angle between the connection line and the normal vector, wherein if the angle is greater than a threshold, the point is a flying pixel point; and both outliers and flying pixel points are boundary points. ### Claim 6 (depends on 5) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 5, wherein in step (4-4), if a triangular patch Δabc in P1 is parallel to a triangular patch Δmnq of P2′ within the neighborhood, then: (I-1) calculating a distance between the Δabc plane and the Δmnq plane; if the distance is greater than a preset moving distance threshold, skipping adjusting Δmnq; if the distance is less than or equal to the preset moving distance threshold, calculating a distance between normals of Δabc and Δmnq that cross the centroids of the triangles; if the distance is greater than a preset threshold, skipping adjusting Δmnq; and if the distance is less than or equal to the preset threshold, calculating a median plane of Δabc and Δmnq, and projecting three vertices of Δmnq onto the median plane along a normal direction of the median plane, to obtain Δm′n′q; and (I-2) iteratively calculating a distance between the Δabc plane and the Δm′n′q′ plane, and repeating step (I-1) until the distance is less than a final distance threshold to obtain a point cloud of a new triangular patch. ### Claim 7 (depends on 5) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 5, wherein in step (4-4), if a triangular patch Δabc in P1 intersects with a triangular patch Δmnq of P2′ within the neighborhood, then: (II-1) calculating an angle between the Δabc plane and the Δmnq plane; if the angle is greater than an initial angle threshold, skipping adjusting Δmnq; if the angle is less than or equal to the initial angle threshold, calculating a median plane of Δabc and Δmnq, and projecting three vertices of Δmnq onto the median plane along a normal direction of the median plane to obtain Δm′n′q; and (II-2) iteratively calculating the angle between the Δabc plane and the Δm′n′q′ plane, and repeating step (II-1) until the angle is less than a final angle threshold, to obtain a point cloud of a new triangular patch. ### Claim 8 (depends on 5) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 5, wherein in step (4-4), if a triangular patch Δabc in P1 intersects with a plane of a triangular patch Δmnq of P2′ within the neighborhood, then: (III-1) calculating an angle between the Δabc plane and the Δmnq plane; if the angle is greater than an initial angle threshold, skipping adjusting Δmnq; if the angle is less than or equal to the initial angle threshold, calculating a median plane of Δabc and Δmnq, and projecting three vertices of Δmnq onto the median plane along a normal direction of the median plane to obtain Δm′n′q; calculating a connection line distance between the projected points m′, n′, and q′ and the origins m, n, and q; if the distance is greater than a preset moving distance threshold, skipping adjusting Δmnq; and if the distance is less than or equal to the preset moving distance threshold, performing step (III-2); and (III-2) iteratively calculating the angle between the Δabc plane and the Δm′n′q′ plane, and repeating step (III-1) until the angle is less than a final angle threshold, to obtain a point cloud of a new triangular patch. ### Claim 9 (depends on 1) The plant point cloud acquisition, registration, and optimization method based on a TOF camera according to claim 1, wherein the step (4-3) of checking whether the triangular patch in P1 intersects with or is in parallel to the triangular patches within the neighborhood comprises: defining Δabc as a triangular patch in P1, and Δmnq as a triangular patch in P2′ within the neighborhood of Δabc; (1) calculating plane equations and normal vectors {right arrow over (n)}1 and {right arrow over (n)}2 of Δabc and Δmnq; (2) when {right arrow over (n)}1·{right arrow over (n)}2−Threshold, Δabc is parallel to Δmnq, wherein Threshold>0, and Threshold is a preset threshold; and and (3) when {right arrow over (n)}1·{right arrow over (n)}2>Threshold and {right arrow over (n)}1·{right arrow over (n)}2<−Threshold, Δabc intersects with Δmnq; in this case, according to the Moller method, determining whether an intersection of a side of Δmnq and the Δabc plane is within Δabc; if the intersection of the side of Δmnq and the Δabc plane is within Δabc, Δabc intersects with Δmnq; otherwise, Δmnq intersects with the Δabc plane.