# US 2023/0274540 A1 · Autonomous Vehicle Lane Boundary Detection Systems and Methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274540 A1
- Document ID: US20230274540A1
- Status: Published application
- Kind: A1
- Application number: 18/313,794
- Filed: 2023-05-08
- Published: 2023-08-31
- Earliest claimed: 2017-11-15
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Min Bai; Gellert Sandor Mattyus; Namdar Homayounfar; Shenlong Wang; Shrinidihi Kowshika Lakshmikanth; Raquel Urtasun
- Web page: https://patentgrep.com/patent/US20230274540A1

## CPC

- G06V 10/82: using neural networks
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain a first type of sensor data (e.g., camera image data) associated with a surrounding environment of an autonomous vehicle and/or a second type of sensor data (e.g., LIDAR data) associated with the surrounding environment of the autonomous vehicle. The computing system can generate overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first and/or second types of sensor data. The computing system can determine one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle and a machine-learned lane boundary detection model.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions that are executable by the one or more processors to cause the autonomous vehicle computing system to perform operations, the operations comprising:
obtaining, from a camera system of the autonomous vehicle, image data descriptive of a portion of a surrounding environment;
obtaining ground height data for the portion of the surrounding environment, wherein the ground height data was generated using a machine-learned model configured to process sensor data and output data indicative of ground height;
associating, using the ground height data, one or more portions of the image data with a lane of a travel way based on a transformation between a reference frame of the camera system and a reference frame of the ground height data; and
based on the association between the one or more portions of the image data and the lane, determining an object within the surrounding environment is within the lane.

### Claim 22 (depends on 21)

The autonomous vehicle control system of claim 21, the operations comprising:
based on determining the object is within the lane, determining a motion of the object.

### Claim 23 (depends on 22)

The autonomous vehicle control system of claim 22, wherein determining the motion comprises:
determining a current speed of the object or a past speed of the object.

### Claim 24 (depends on 21)

The autonomous vehicle control system of claim 21, wherein the transformation is based on a projection from the reference frame of the ground height data to the reference frame of the camera system.

### Claim 25 (depends on 21)

The autonomous vehicle control system of claim 21, wherein the ground height data is estimated ground height data generated based on LIDAR data.

### Claim 26 (depends on 21)

The autonomous vehicle control system of claim 21, the operations comprising:
obtaining map data descriptive of a location of the lane; and
generating perception data based on the map data.

### Claim 27 (depends on 26)

The autonomous vehicle control system of claim 26, the operations comprising:
obtaining sensor data comprising the image data and LIDAR data; and
processing the sensor data and the map data to generate the perception data.

### Claim 28

One or more tangible, non-transitory, computer readable media that store instructions that are executable by one or more processors to cause an autonomous vehicle computing system to perform operations, the operations comprising:
obtaining, from a camera system of an autonomous vehicle, image data descriptive of a portion of a surrounding environment;
obtaining ground height data for the portion of the surrounding environment, wherein the ground height data was generated using a machine-learned model configured to process sensor data and output data indicative of ground height;
associating, using the ground height data, one or more portions of the image data with a lane of a travel way based on a transformation between a reference frame of the camera system and a reference frame of the ground height data; and
based on the association between the one or more portions of the image data and the lane, determining an object within the surrounding environment is within the lane.

### Claim 29 (depends on 28)

The one or more computer readable media of claim 28, the operations comprising:
based on determining the object is within the lane, determining a motion of the object.

### Claim 30 (depends on 29)

The one or more computer readable media of claim 29, wherein determining the motion comprises:
determining a current speed of the object or a past speed of the object.

### Claim 31 (depends on 28)

The one or more computer readable media of claim 28, wherein the transformation is based on a projection from the reference frame of the ground height data to the reference frame of the camera system.

### Claim 32 (depends on 28)

The one or more computer readable media of claim 28, wherein the ground height data is estimated ground height data generated based on LIDAR data.

### Claim 33 (depends on 28)

The one or more computer readable media of claim 28, the operations comprising:
obtaining map data descriptive of a location of the lane; and
generating perception data based on the map data.

### Claim 34 (depends on 33)

The one or more computer readable media of claim 33, the operations comprising:
obtaining sensor data comprising the image data and LIDAR data; and
processing the sensor data and the map data to generate the perception data.

### Claim 35

A computer-implemented method comprising:
obtaining, from a camera system of an autonomous vehicle, image data descriptive of a portion of a surrounding environment;
obtaining ground height data for the portion of the surrounding environment, wherein the ground height data was generated using a machine-learned model configured to process sensor data and output data indicative of ground height;
associating, using the ground height data, one or more portions of the image data with a lane of a travel way based on a transformation between a reference frame of the camera system and a reference frame of the ground height data; and
based on the association between the one or more portions of the image data and the lane, determining an object within the surrounding environment is within the lane.

### Claim 36 (depends on 35)

The computer-implemented method of claim 35, comprising:
based on determining the object is within the lane, determining a motion of the object.

### Claim 37 (depends on 36)

The computer-implemented method of claim 36, wherein determining the motion comprises:
determining a current speed of the object or a past speed of the object.

### Claim 38 (depends on 35)

The computer-implemented method of claim 35, wherein the transformation is based on a projection from the reference frame of the ground height data to the reference frame of the camera system.

### Claim 39 (depends on 35)

The computer-implemented method of claim 35, wherein the ground height data is estimated ground height data generated based on LIDAR data.

### Claim 40 (depends on 35)

The computer-implemented method of claim 35, comprising:
obtaining sensor data comprising the image data and LIDAR data;
obtaining map data descriptive of a location of the lane; and
processing the sensor data and the map data to generate perception data.
