# US 11,999,352 B2 · Method and system for extracting road data and method and system for controlling self-driving car > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,999,352 B2 - Document ID: US11999352B2 - Status: Granted - Kind: B2 - Application number: 17/552,177 - Filed: 2021-12-15 - Published: 2023-06-15 - Granted: 2024-06-04 - Earliest claimed: 2021-12-15 - Assignees: Industrial Technology Research Institute - Applicants: Industrial Technology Research Institute - Inventors: Victor Lu; Chun-Jung Lin; KunLung Ku - Web page: https://patentgrep.com/patent/US11999352B2 ## CPC - B60W 40/06: Road conditions - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - B62D 6/002: {computing target steering angles for front or rear wheels (B62D7/159 takes precedence)} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60W 2552/15: Road slope, i.e. the inclination of a road segment in the longitudinal direction - B60W 2552/30: Road curve radius - B60W 2552/35: Road bumpiness, e.g. potholes - B62D 15/025: {Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation (B62D1/28 takes precedence)} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for extracting road data comprises obtaining ground data from first environment detection data of a detected region. A first ground model may be obtained using the ground data. A missing region of the first ground model may be filled to obtain a second ground model. Road attributes of the detected region may be obtained according to the second ground model. Road damage recognition may be performed using second environment detection data of the detected region and the second ground model. The method may include storing the road attributes and the result of the road damage recognition as pieces of road data of the detected region. ## Claims ### Claim 1 A method for extracting road data, comprising performing following steps by a processing device: obtaining ground data from first environment detection data of a detection region, wherein the first environment detection data is a point cloud; obtaining a first ground model using the ground data; filling in missing regions of the first ground model to obtain a second ground model; obtaining a plurality of road attributes of the detection region according to the second ground model; performing road damage identification using second environment detection data of the detection region and the second ground model; and storing the plurality of road attributes and a result of the road damage identification as a plurality of pieces of road data of the detection region, wherein obtaining the ground data from the first environment detection data of the detection region comprises: dividing the point cloud into a plurality of original voxels, wherein the plurality of original voxels comprise a plurality of voxel columns; obtaining a plurality of target voxel groups from the plurality of original voxels, wherein the plurality of target voxel groups are respectively included in a plurality of voxel columns, a vertical spread parameter of a plurality of points in each of the plurality of target voxel groups is less than or equal to a preset value, and the vertical spread parameter is a height difference between a highest point and a lowest point in each of the plurality of target voxel groups; calculating a plurality of height values of the plurality of target voxel groups; and generating the ground data using the plurality of height values. ### Claim 2 (depends on 1) The method for extracting road data according to claim 1, wherein each of the plurality of voxel columns extends along a vertical axis; each of the plurality of target voxel groups is a group of vertically contiguous voxels satisfying a condition that each of the plurality of target voxel groups resides in a corresponding one of the plurality of voxel column, a surface normal of a surface fitted to the plurality of points in each of the plurality of target voxel groups is upward; each height value represents an average height of the plurality of points in a target voxel group. ### Claim 3 (depends on 2) The method for extracting road data according to claim 2, wherein the surface normal is determined by principal component analysis and a scanner trajectory. ### Claim 4 (depends on 2) The method for extracting road data according to claim 2, wherein generating the ground data using the plurality of height values comprises: performing a region growing operation or a connected component operation on the plurality of height values to form a plurality of connected groups of height values; and selecting at least one of the plurality of connected groups of height values as the ground data by selecting one or more connected groups having a sufficiently large number of height values and having sufficient overlap with a scanner trajectory. ### Claim 5 (depends on 1) The method for extracting road data according to claim 1, wherein filling in the missing region of the first ground model to obtain the second ground model comprises: transforming the first ground model into two-dimensional data comprising a plurality of pixels respectively storing the plurality of height values of the first ground model; performing digital inpainting on the two-dimensional data to generate repaired two-dimensional data; and transforming the repaired two-dimensional data into a three-dimensional model as the second ground model. ### Claim 6 (depends on 1) The method for extracting road data according to claim 1, wherein the second environment detection data is a point cloud and the second ground model comprises an original curved surface, and the performing road damage identification using second environment detection data of the detection region and the second ground model comprises: shifting down the original curved surface with a first distance along a vertical axis to form a first curved surface; shifting down the original curved surface with a second distance greater than the first distance along the vertical axis to form a second curved surface; and forming one or more damage regions by identifying a plurality of points from the second environment detection data, wherein the plurality of points from the second environment detection data is in between the first curved surface and the second curved surface. ### Claim 7 (depends on 6) The method for extracting road data according to claim 6, wherein the first distance is associated with a precision of the point cloud, and the second distance is associated with a detection error of an environment detector generating the point cloud, including errors due to environment properties, such as reflective wet road surfaces. ### Claim 8 (depends on 1) The method for extracting road data according to claim 1, wherein the plurality of road attributes are stored in a format of one of a triangular irregular network, a regular grid and waypoints. ### Claim 9 (depends on 1) The method for extracting road data according to claim 1, wherein the first environment detection data and the second environment detection data are detected at different times. ### Claim 10 A method for controlling a self-driving car comprising performing following steps by a processing device: predicting the self-driving car's position at time of actuation of a given chassis control command, using an estimate of a processing delay time, an estimate of a chassis control delay time, a current position and a current velocity of the self-driving car; obtaining target road data corresponding to a estimated position from a map database; and generating a chassis control command according to the target road data to control the self-driving car; wherein the map database comprises a plurality of pieces of pre-stored road data of a detection region, the estimated position is in the detection region, the target road data is included in the plurality of pieces of pre-stored road data and the plurality of pieces of pre-stored road data are obtained by a road data extracting procedure, and the road data extracting procedure comprises: obtaining ground data from first environment detection data of the detection region, wherein the first environment detection data is a point cloud; obtaining a first ground model using the ground data; filling in missing regions of the first ground model to obtain a second ground model; obtaining a plurality of road attributes of the detection region according to the second ground model; performing road damage identification using second environment detection data of the detection region and the second ground model; and storing the plurality of road attributes and a result of the road damage identification as a plurality of pieces of pre-stored road data of the detection region, wherein obtaining the ground data from the first environment detection data of the detection region comprises: dividing the point cloud into a plurality of original voxels, wherein the plurality of original voxels comprise a plurality of voxel columns; obtaining a plurality of target voxel groups from the plurality of original voxels, wherein the plurality of target voxel groups are respectively included in a plurality of voxel columns, a vertical spread parameter of a plurality of points in each of the plurality of target voxel groups is less than or equal to a preset value, and the vertical spread parameter is a height difference between a highest point and a lowest point in each of the plurality of target voxel groups; calculating a plurality of height values of the plurality of target voxel groups; and generating the ground data using the plurality of height values. ### Claim 11 (depends on 10) The method for controlling self-driving car according to claim 10, wherein predicting the self-driving car's position at time of actuation of a given chassis control command, using an estimate of the processing delay time, an estimate of chassis control delay time, a current position and a current velocity of the self-driving car, comprises: adding the processing delay time and the chassis control delay time to obtain an estimate of total delay time; multiplying the total delay time by the current velocity to obtain an estimated displacement vector; and obtaining the estimated position using the current position and the estimated displacement vector. ### Claim 12 A system for extracting road data comprising: a data input device configured to receive first environment detection data and second environment detection data, wherein the first environment detection data is a point cloud; a storage device; and a processing device connected to the data input device and the storage device and configured to perform following steps: obtaining ground data from the first environment detection data; obtaining a first ground model using the ground data; filling in missing regions of the first ground model to obtain a second ground model; obtaining a plurality of road attributes of a detection region according to the second ground model; performing road damage identification using the second environment detection data and the second ground model; and storing the plurality of road attributes and a result of the road damage identification as a plurality of pieces of road data of the detection region, wherein obtaining the ground data from the first environment detection data of the detection region comprises: dividing the point cloud into a plurality of original voxels, wherein the plurality of original voxels comprise a plurality of voxel columns; obtaining a plurality of target voxel groups from the plurality of original voxels, wherein the plurality of target voxel groups are respectively included in a plurality of voxel columns, a vertical spread parameter of a plurality of points in each of the plurality of target voxel groups is less than or equal to a preset value, and the vertical spread parameter is a height difference between a highest point and a lowest point in each of the plurality of target voxel groups; calculating a plurality of height values of the plurality of target voxel groups; and generating the ground data using the plurality of height values. ### Claim 13 (depends on 12) The system for extracting road data according to claim 12, wherein each of the plurality of voxel columns extends along a vertical axis, each of the plurality of target voxel groups is a group of vertically contiguous voxels satisfying a condition that each of the plurality of target voxel groups resides in a corresponding one of the plurality of voxel column and a surface normal of a surface fitted to the plurality of points in each of the plurality of target voxel groups is upward, and each of plurality of height values represents an average height of the plurality of points in each of the plurality of target voxel groups. ### Claim 14 (depends on 13) The system for extracting road data according to claim 13, wherein the surface normal is determined by principal component analysis and a scanner trajectory. ### Claim 15 (depends on 13) The system for extracting road data according to claim 13, wherein the processing device is configured to perform a region growing operation or a connected component operation on the plurality of height values to form a plurality of connected groups of height values and select at least one of the plurality connected groups of height values as the ground data according to the size of a connected group and the amount of overlap between a connected group of height values and a scanner trajectory. ### Claim 16 (depends on 12) The system for extracting road data according to claim 12, wherein the processing device is configured to transform the first ground model into two-dimensional data, performs digital inpainting on the two-dimensional data to generate repaired two-dimensional data and transform the repaired two-dimensional data into a three-dimensional model as the second ground model; wherein the two-dimensional data comprises a plurality of pixels respectively storing the plurality of height values of the first ground model. ### Claim 17 (depends on 12) The system for extracting road data according to claim 12, wherein the second environment detection data is a point cloud and the second ground model comprises an original curved surface, and the processing device shifts down the original curved surface with a first distance along a vertical axis to form a first curved surface, and shifts down the original curved surface with a second distance greater than the first distance along the vertical axis to form a second curved surface and determines a plurality of points from the second environment detection data between the first curved surface and the second curved surface in the point cloud to form one or more damage regions. ### Claim 18 (depends on 17) The system for extracting road data according to claim 17, wherein the first distance is associated with a precision of the point cloud, and the second distance is associated with a detection errors due to environment factors, including wet reflective road surfaces. ### Claim 19 (depends on 12) The system for extracting road data according to claim 12, wherein the plurality of road attributes are stored in a format of one of a triangular irregular network, a regular grid and waypoints. ### Claim 20 (depends on 12) The system for extracting road data according to claim 12, wherein the first environment detection data and the second environment detection data are detected at different times. ### Claim 21 A system for controlling a self-driving car comprising: a positioning module configured to obtain a current position of the self-driving car; a car dynamic state sensing module configured to obtain a current velocity of the self-driving car; a chassis control module configured to control the self-driving car according to a chassis control command; a processing module connected to the positioning module, the car dynamic state sensing module and the chassis control module and configured to perform following steps: predicting the self-driving car's estimated position at time of actuation of a given chassis control command, using the self-driving car's current position, the car's current velocity, an estimate of processing delay time and an estimate of chassis control delay time; obtaining target road data corresponding to the estimated position from a map database; and generating a chassis control command, according to the target road data to control the self-driving car; wherein the map database comprises a plurality of pieces of pre-stored road data of a detection region, the estimated position is in the detection region, and the target road data is included in the plurality of pieces of pre-stored road data and the plurality of pieces of pre-stored road data are obtained by a road data extracting procedure, and the road data extracting procedure comprises: obtaining ground data from first environment detection data of a detection region, wherein the first environment detection data is a point cloud; obtaining a first ground model using the ground data; filling in missing regions of the first ground model to obtain a second ground model; obtaining a plurality of road attributes of the detection region according to the second ground model; performing road damage identification using second environment detection data of the detection region and the second ground model; and storing the plurality of road attributes and a result of the road damage identification as a plurality of pieces of pre-stored road data of the detection region, wherein obtaining the ground data from the first environment detection data of the detection region comprises: dividing the point cloud into a plurality of original voxels, wherein the plurality of original voxels comprise a plurality of voxel columns; obtaining a plurality of target voxel groups from the plurality of original voxels, wherein the plurality of target voxel groups are respectively included in a plurality of voxel columns, a vertical spread parameter of a plurality of points in each of the plurality of target voxel groups is less than or equal to a preset value, and the vertical spread parameter is a height difference between a highest point and a lowest point in each of the plurality of target voxel groups; calculating a plurality of height values of the plurality of target voxel groups; and generating the ground data using the plurality of height values. ### Claim 22 (depends on 21) The system for controlling self-driving car according to claim 21, wherein the processing module adds the processing time of the processing module and the chassis control delay time to obtain a total delay time, multiplies the total delay time by the current velocity to obtain an estimated displacement vector and obtains the estimated position using the current position and the estimated displacement vector.