# US 12,112,552 B2 · Lane marker recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,552 B2
- Document ID: US12112552B2
- Status: Granted
- Kind: B2
- Application number: 17/655,500
- Filed: 2022-03-18
- Published: 2023-09-21
- Granted: 2024-10-08
- Earliest claimed: 2022-03-18
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Seungwoo Yoo; Heesoo Myeong; Hee-Seok Lee
- Web page: https://patentgrep.com/patent/US12112552B2

## CPC

- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06T 7/73: using feature-based methods
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/507: {Summing image-intensity values; Histogram projection analysis}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30256: Lane; Road marking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Certain aspects of the present disclosure provide techniques for lane marker detection. A set of feature tensors is generated by processing an input image using a convolutional neural network. A set of localizations is generated by processing the set of feature tensors using a localization network, a set of horizontal positions is generated by processing the set of feature tensors using row-wise regression, and a set of end positions is generated by processing the set of feature tensors using y-end regression. A set of lane marker positions is determined based on the set of localizations, the set of horizontal positions, and the set of end positions.

## Claims

### Claim 1

A computer-implemented method, comprising:
generating a set of feature tensors by processing an input image using a convolutional neural network;
generating a set of localizations by processing the set of feature tensors using a localization network;
generating a set of horizontal positions by processing the set of feature tensors using row-wise regression;
generating a set of end positions by processing the set of feature tensors using y-end regression; and
determining a set of lane marker positions based on the set of localizations, the set of horizontal positions, and the set of end positions.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the set of lane marker positions comprises:
identifying a first subset of localization positions, from the set of localizations that exceeds a defined threshold; and
for each respective localization of the first subset of localization positions, selecting a corresponding horizontal position of the set of horizontal positions based on a respective end position of the set of end positions.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the set of lane marker positions comprises:
selecting a second subset from the first subset of localization positions using non-maximum suppression based on distance between respective localizations in the first subset of localization positions, wherein the distance is defined based on horizontal distance between overlapping regions.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
applying a random sample consensus (RANSAC) technique to the set of lane marker positions; and
fitting the set of lane marker positions to one or more polynomial curves.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
the convolutional neural network is a feature pyramid network comprising a bottom-up pathway and a top-down pathway connected by one or more lateral connections; and
prior to generating the set of localizations, the set of horizontal positions, and the set of end positions, the set of feature tensors are aggregated using a set of learned weights.

### Claim 6 (depends on 5)

The method of claim 5, wherein the set of learned weights comprise:
a first weight corresponding to the localization network;
a second weight corresponding to the row-wise regression; and
a third weight corresponding to the y-end regression.

### Claim 7 (depends on 1)

The method of claim 1, wherein training the localization network comprises computing a cross-entropy loss based on output of the localization network and a ground-truth location of one or more lane markers, wherein the ground-truth location comprises a center point and at least one end point.

### Claim 8 (depends on 1)

The method of claim 1, wherein the row-wise regression comprises a softmax function, and wherein training the row-wise regression comprises computing a smooth L1 loss based on output of the softmax function of the row-wise regression.

### Claim 9 (depends on 8)

The method of claim 8, wherein training the row-wise regression further comprises computing a confidence loss using a dice loss function.

### Claim 10 (depends on 1)

The method of claim 1, wherein the y-end regression comprises an exponential function, and wherein training the y-end regression comprises computing a smooth L1 loss based on output of the exponential function of the y-end regression.

### Claim 11

A processing system, comprising:
a memory comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the processing system to perform an operation comprising:
generating a set of feature tensors by processing an input image using a convolutional neural network;
generating a set of localizations by processing the set of feature tensors using a localization network;
generating a set of horizontal positions by processing the set of feature tensors using row-wise regression;
generating a set of end positions by processing the set of feature tensors using y-end regression; and
determining a set of lane marker positions based on the set of localizations, the set of horizontal positions, and the set of end positions.

### Claim 12 (depends on 11)

The processing system of claim 11, wherein determining the set of lane marker positions comprises:
identifying a first subset of localization positions, from the set of localizations that exceeds a defined threshold; and
for each respective localization of the first subset of localization positions, selecting a corresponding horizontal position of the set of horizontal positions based on a respective end position of the set of end positions.

### Claim 13 (depends on 12)

The processing system of claim 12, wherein determining the set of lane marker positions comprises:
selecting a second subset from the first subset of localization positions using non-maximum suppression based on distance between respective localizations in the first subset of localization positions, wherein the distance is defined based on horizontal distance between overlapping regions.

### Claim 14 (depends on 11)

The processing system of claim 11, the operation further comprising:
applying a random sample consensus (RANSAC) technique to the set of lane marker positions; and
fitting the set of lane marker positions to one or more polynomial curves.

### Claim 15 (depends on 11)

The processing system of claim 11, wherein:
the convolutional neural network is a feature pyramid network comprising a bottom-up pathway and a top-down pathway connected by one or more lateral connections; and
prior to generating the set of localizations, the set of horizontal positions, and the set of end positions, the set of feature tensors are aggregated using a set of learned weights.

### Claim 16 (depends on 15)

The processing system of claim 15, wherein the set of learned weights comprise:
a first weight corresponding to the localization network;
a second weight corresponding to the row-wise regression; and
a third weight corresponding to the y-end regression.

### Claim 17 (depends on 11)

The processing system of claim 11, wherein training the localization network comprises computing a cross-entropy loss based on output of the localization network and a ground-truth location of one or more lane markers, wherein the ground-truth location comprises a center point and at least one end point.

### Claim 18 (depends on 11)

The processing system of claim 11, wherein the row-wise regression comprises a softmax function, and wherein training the row-wise regression comprises computing a smooth L1 loss based on output of the softmax function of the row-wise regression.

### Claim 19 (depends on 18)

The processing system of claim 18, wherein training the row-wise regression further comprises computing a confidence loss using a dice loss function.

### Claim 20 (depends on 11)

The processing system of claim 11, wherein the y-end regression comprises an exponential function, and wherein training the y-end regression comprises computing a smooth L1 loss based on output of the exponential function of the y-end regression.

### Claim 21

A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform an operation comprising:
generating a set of feature tensors by processing an input image using a convolutional neural network;
generating a set of localizations by processing the set of feature tensors using a localization network;
generating a set of horizontal positions by processing the set of feature tensors using row-wise regression;
generating a set of end positions by processing the set of feature tensors using y-end regression; and
determining a set of lane marker positions based on the set of localizations, the set of horizontal positions, and the set of end positions.

### Claim 22 (depends on 21)

The processing system of claim 21, wherein determining the set of lane marker positions comprises:
identifying a first subset of localization positions, from the set of localizations that exceeds a defined threshold; and
for each respective localization of the first subset of localization positions, selecting a corresponding horizontal position of the set of horizontal positions based on a respective end position of the set of end positions.

### Claim 23 (depends on 22)

The processing system of claim 22, wherein determining the set of lane marker positions comprises:
selecting a second subset from the first subset of localization positions using non-maximum suppression based on distance between respective localizations in the first subset of localization positions, wherein the distance is defined based on horizontal distance between overlapping regions.

### Claim 24 (depends on 21)

The processing system of claim 21, the operation further comprising:
applying a random sample consensus (RANSAC) technique to the set of lane marker positions; and
fitting the set of lane marker positions to one or more polynomial curves.

### Claim 25 (depends on 21)

The processing system of claim 21, wherein:
the convolutional neural network is a feature pyramid network comprising a bottom-up pathway and a top-down pathway connected by one or more lateral connections; and
prior to generating the set of localizations, the set of horizontal positions, and the set of end positions, the set of feature tensors are aggregated using a set of learned weights.

### Claim 26 (depends on 25)

The processing system of claim 25, wherein the set of learned weights comprise:
a first weight corresponding to the localization network;
a second weight corresponding to the row-wise regression; and
a third weight corresponding to the y-end regression.

### Claim 27 (depends on 21)

The processing system of claim 21, wherein training the localization network comprises computing a cross-entropy loss based on output of the localization network and a ground-truth location of one or more lane markers, wherein the ground-truth location comprises a center point and at least one end point.

### Claim 28 (depends on 21)

The processing system of claim 21, wherein the row-wise regression comprises a softmax function, and wherein training the row-wise regression comprises computing a smooth L1 loss based on output of the softmax function of the row-wise regression.

### Claim 29 (depends on 28)

The processing system of claim 28, wherein training the row-wise regression further comprises computing a confidence loss using a dice loss function.

### Claim 30 (depends on 21)

The processing system of claim 21, wherein the y-end regression comprises an exponential function, and wherein training the y-end regression comprises computing a smooth L1 loss based on output of the exponential function of the y-end regression.