# US 10,373,003 B2 · Deep module and fitting module system and method for motion-based lane detection with multiple sensors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,003 B2
- Document ID: US10373003B2
- Status: Granted
- Kind: B2
- Application number: 15/683,463
- Filed: 2017-08-22
- Published: 2019-02-28
- Granted: 2019-08-06
- Earliest claimed: 2017-08-22
- Assignees: Tusimple, Inc.
- Applicants: Tusimple
- Inventors: Siyuan Liu; Mingdong Wang; Xiaodi Hou
- Web page: https://patentgrep.com/patent/US10373003B2

## CPC

- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60W 40/072: Curvature of the road
- G01S 19/01: Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- G06T 2207/30256: Lane; Road marking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of lane detection for a non-transitory computer readable storage medium storing one or more programs is disclosed. The one or more programs include instructions, which when executed by a computing device, cause the computing device to perform the following steps comprising: generating a ground truth; off-line training a lane detection algorithm by using the ground truth, the lane detection algorithm using parameters that express a lane marking in an arc; on-line generating a predicted lane marking; comparing the predicted lane marking against the ground truth; and off-line refining the lane detection algorithm.

## Claims

### Claim 1

A method of lane detection for a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the following steps comprising:
generating a ground truth;
off-line training a lane detection algorithm by using the ground truth, the lane detection algorithm using parameters that express a lane marking in an arc;
on-line generating a predicted lane marking for a current view;
comparing the predicted lane marking against the ground truth; and
off-line refining the lane detection algorithm by using the lane template associated with the current view to generate an improved lane template used for improved lane detection of a next view;
wherein, a detected lane map is generated relative to a moving vehicle using the improved lane template and the predicted lane marking.

### Claim 2 (depends on 1)

The method according to claim 1, wherein generating a ground truth includes:
collecting data in an environment by using sensors that include an inertial measurement unit (IMU) module, a global positioning system (GPS) module and a mapping (MAP) module; and
generating a labeled lane marking by annotating a lane marking expressed in god's view.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the IMU module provides information on at least one of a vehicle pose or a vehicle speed, and the global positioning system (GPS) module provides global position information.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the MAP module is configured to create a map of its surroundings, and orient a vehicle itself within this map.

### Claim 5 (depends on 1)

The method according to claim 1, wherein on-line generating a predicted lane marking includes:
generating a hit-map image for the current view based on the lane detection algorithm; and
generating a fitted lane marking based on the hit-map image and a lane template that includes features of a view immediately previous to the current view.

### Claim 6 (depends on 5)

The method according to claim 5, wherein generating a fitted lane marking includes:
optimizing, based on priors or constraints, the lane template to obtain a local optimal.

### Claim 7 (depends on 5)

The method according to claim 5 further comprising:
determining that a confidence level of the fitted lane marking is reasonable, using the parameters; and
outputting the fitted lane marking as a predicted lane marking.

### Claim 8 (depends on 5)

The method according to claim 5 further comprising:
determining that a confidence level of the fitted lane marking is unreasonable, using the parameters; and
rejecting the fitted lane marking.

### Claim 9 (depends on 1)

The method according to claim 1, wherein off-line refining the lane detection algorithm includes:
adding additional ground truth data in the off-line training.

### Claim 10 (depends on 1)

The method according to claim 1 further comprising:
on-line generating another predicted lane marking, using a refined lane detection algorithm.

### Claim 11

A system for lane detection, the system comprising:
an internet server, comprising:
an I/O port, configured to transmit and receive electrical signals to and from a client device;
a memory;
one or more processing units; and
one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions for:
generating a ground truth;
off-line training a lane detection algorithm by using the ground truth, the lane detection algorithm using parameters that express a lane marking in an arc;
on-line generating a predicted lane marking for a current view;
comparing the predicted lane marking against the ground truth; and
off-line refining the lane detection algorithm by using the lane template associated with the current view to generate an improved lane template used for improved lane detection of a next view;
wherein, a detected lane map is generated relative to a moving vehicle using the improved lane template and the predicted lane marking.

### Claim 12 (depends on 11)

The system according to claim 11, wherein generating a ground truth includes:
collecting data in an environment by using sensors that include an inertial measurement unit (IMU) module, a global positioning system (GPS) module and a mapping (MAP) module; and
generating a labeled lane marking by annotating a lane marking expressed in god's view.

### Claim 13 (depends on 12)

The system according to claim 12, wherein the IMU module provides information on at least one of a vehicle pose or a vehicle speed, and the global positioning system (GPS) module provides global position information.

### Claim 14 (depends on 12)

The system according to claim 12, wherein the MAP module is configured to create a map of its surroundings, and orient a vehicle itself within this map.

### Claim 15 (depends on 11)

The system according to claim 11, wherein on-line generating a predicted lane marking includes:
generating a hit-map image for the current view based on the lane detection algorithm; and
generating a fitted lane marking based on the hit-map image and a lane template that includes features of a view immediately previous to the current view.

### Claim 16 (depends on 15)

The system according to claim 15, wherein generating a fitted lane marking includes:
optimizing, based on priors or constraints, the lane template to obtain a local optimal.

### Claim 17 (depends on 15)

The system according to claim 15 further comprising:
determining that a confidence level of the fitted lane is reasonable, using the parameters; and
outputting the fitted lane marking as a predicted lane marking.

### Claim 18 (depends on 15)

The system according to claim 15 further comprising:
determining that a confidence level of the fitted lane is unreasonable, using the parameters; and
rejecting the fitted lane marking.

### Claim 19 (depends on 11)

The system according to claim 11, wherein off-line refining the lane detection algorithm includes:
adding additional ground truth data in the off-line training.

### Claim 20 (depends on 11)

The system according to claim 11 further comprising:
on-line generating another predicted lane marking, using a refined lane detection algorithm.