# US 11,113,546 B2 · Lane line processing method and device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,546 B2 - Document ID: US11113546B2 - Status: Granted - Kind: B2 - Application number: 16/521,028 - Filed: 2019-07-24 - Published: 2020-03-05 - Granted: 2021-09-07 - Earliest claimed: 2018-09-04 - Assignees: Baidu Online Network Technology (Beijing) Co., Ltd. - Applicants: Baidu Online Network Technology (Beijing) Co., Ltd. - Inventors: Shufu Xie; Yuqiang Zhai; Tian Xia; Yu Ma - Web page: https://patentgrep.com/patent/US11113546B2 ## CPC - G06T 7/13: Edge detection - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06N 20/00: Machine learning - G06T 7/181: involving edge growing; involving edge linking - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A lane line processing method and device are provided. The method can includes obtaining distances between lane line points in a first image, determining direction densities of the individual lane line points by using the distances between the lane line points, dividing lane line points into groups corresponding to lane lines by using the direction densities of the individual lane line points, and obtaining representation information of the lane lines corresponding to the groups by using the lane line points in the groups. By using the direction densities of the individual lane line points, lane line points in an image are divided into groups, the obtained groups are then more accurate, and thus the lane lines obtained by fitting are accurate, are less susceptible to image quality issues, and have high robustness. ## Claims ### Claim 1 A lane line processing method, comprising: obtaining distances between lane line points in a first image; determining direction densities of the individual lane line points by using the distances between the lane line points, each direction density indicating the probability that an individual lane line point approaches other lane line points in a direction; dividing the lane line points into groups corresponding to lane lines by using the direction densities of the individual lane line points; and obtaining representation information of the lane lines corresponding to the groups by using the lane line points in the groups. ### Claim 2 (depends on 1) The method of claim 1, further comprising: inputting the first image to a deep learning model to obtain a second image, wherein there is a scaling ratio between the first image and the second image, and wherein the second image comprises lane line points and non-lane line points obtained by identifying the first image in the deep learning model; and mapping coordinates of the lane line points in the second image back to the first image according to the scaling ratio, to obtain coordinates of the lane line points in the first image. ### Claim 3 (depends on 1) The method of claim 1, wherein determining direction densities of the individual lane line points by using the distances between the lane line points comprises: selecting a target lane line point and K lane line points, wherein a distance from each of the K lane line points to the target lane line point is less than a first threshold, K is a positive integer greater than or equal to 1; generating a plurality of straight lines, wherein each of the straight lines passes through the target lane line point and at least one of the K lane line points; obtaining, for a respective straight line, the number of relevant lane line points, wherein a distance from each relevant lane line point to the respective straight line is less than a second threshold; and determining a maximum of numbers of the relevant lane line points corresponding to the respective straight line, as a direction density of the target lane line point. ### Claim 4 (depends on 1) The method of claim 1, wherein dividing the lane line points into groups corresponding to lane lines by using the direction densities of the individual lane line points comprises: determining a candidate lane line point of a selected group of the groups; determining, in the first image, a lane line point belonging to the selected group, according to the direction densities of the individual lane line points; adding the determined lane line point into the selected group; and taking the added lane line point as a new candidate lane line point, and continuing the determining a lane line point belonging to the selected group, until no lane line point belonging to the selected group, is determined; wherein determining, in the first image, a lane line point belonging to the selected group, according to the direction densities of the individual lane line points comprises: selecting M lane line points from the first image, wherein a distance from each of the M lane line points to the candidate lane line point is less than a third threshold, M is a positive integer greater than or equal to 1; and determining lane line points belonging to the selected group according to the direction densities of the M lane line points. ### Claim 5 (depends on 4) The method of claim 4, wherein determining, in the first image, a lane line point belonging to the selected group, according to the direction densities of the M lane line points comprises: determining a lane line point having the maximum direction density among the M lane line points as the lane line point belonging to the selected group; or obtaining, from the M lane line points, relevant lane line points of a straight line corresponding to a direction density of the candidate lane line point, and selecting a point having the maximum direction density from the obtained relevant lane line points as the lane line point belonging to the selected group. ### Claim 6 (depends on 1) The method of claim 1, wherein obtaining representation information of the lane lines corresponding to the groups by using the lane line points in the groups comprises: selecting, for each group, a plurality of lane line points; and performing polynomial fitting on coordinates of the selected plurality of lane line points in the first image, to obtain a polynomial curve of a lane line corresponding to the group. ### Claim 7 (depends on 6) The method of claim 6, wherein obtaining representation information of the lane lines corresponding to the groups by using the lane line points in the groups further comprises: determining coordinates of start points and/or coordinates of end points of the lane line by using the polynomial curve of the lane line. ### Claim 8 (depends on 1) The method of claim 1, further comprising: merging the lane line points in one of the groups. ### Claim 9 (depends on 1) The method of claim 1, wherein obtaining distances between lane line points in a first image comprises: calculating distances between lane line points according to coordinates of the lane line points in the first image; and generating a distance matrix by using the distances between lane line points, wherein an element in the distance matrix represents a distance between two lane line points in the first image. ### Claim 10 (depends on 1) A non-volatile computer readable storage medium having computer programs stored thereon, wherein the programs, when executed by a processor, cause the processor to implement the method of claim 1. ### Claim 11 A lane line processing device, comprising: one or more processors; a storage device for storing one or more programs; wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: obtain distances between lane line points in a first image; determine direction densities of the individual lane line points by using the distances between the lane line points, each direction density indicating the probability that an individual lane line point approaches other lane line points in a direction; divide lane line points into groups corresponding to lane lines by using the direction densities of the individual lane line points; and obtain representation information of the lane lines corresponding to the groups by using the lane line points in the groups. ### Claim 12 (depends on 11) The device of claim 11, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: input the first image to a deep learning model to obtain a second image, wherein there is a scaling ratio between the first image and the second image, and wherein the second image comprises lane line points and non-lane line points obtained by identifying the first image in the deep learning model; and map coordinates of the lane line points in the second image back to the first image according to the scaling ratio, to obtain coordinates of the lane line points in the first image. ### Claim 13 (depends on 12) The device of claim 12, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: select a target lane line point and K lane line points, wherein a distance from each of the K lane line points to the target lane line point is less than a first threshold, K is a positive integer greater than or equal to 1; generate a plurality of straight lines, wherein each of the straight lines passes through the target lane line point and at least one of the K lane line points; obtain, for a respective straight line, the number of relevant lane line points, wherein a distance from each relevant lane line point to the respective straight line is less than a second threshold; and determine a maximum of numbers of the relevant lane line points corresponding to the respective straight line, as a direction density of the target lane line point. ### Claim 14 (depends on 13) The device of claim 13, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: determine a candidate lane line point of a selected group of the groups; determine, in the first image, a lane line point belonging to the selected group, according to the direction densities of the individual lane line points; add the determined lane line point into the selected groups; and take the added lane line point as a new candidate lane line point, and continue the determining a lane line point belonging to the selected group, until no lane line point belonging to the selected group is determined; wherein the one or more programs are executed by the one or more processors to enable the one or more processors further to: select M lane line points from the first image, wherein a distance from each of the M lane line points to the candidate lane line point is less than a third threshold, M is a positive integer greater than or equal to 1; and determine lane line points belonging to the selected group according to the direction densities of the M lane line points. ### Claim 15 (depends on 11) The device of claim 11, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: select, for each group, a plurality of lane line points; and perform polynomial fitting on coordinates of the selected plurality of lane line points in the first image, to obtain a polynomial curve of a lane line corresponding to the group. ### Claim 16 (depends on 15) The device of claim 15, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: determine coordinates of start points and/or coordinates of end points of the lane line by using the polynomial curve of the lane line. ### Claim 17 (depends on 11) The device of claim 11, wherein the one or more programs are executed by the one or more processors to enable the one or more processors to: calculate distances between lane line points according to coordinates of the lane line points in the first image; and generate a distance matrix by using the distances between lane line points, wherein an element in the distance matrix represents a distance between two lane line points in the first image.