# US 2021/0343150 A1 · Traffic light detection and lane state recognition for autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343150 A1
- Document ID: US20210343150A1
- Status: Published application
- Kind: A1
- Application number: 17/336,856
- Filed: 2021-06-02
- Published: 2021-11-04
- Earliest claimed: 2018-10-24
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Maxim Krivokon; Abhijit S. Ogale; Edward Hsiao; Andreas Wendel
- Web page: https://patentgrep.com/patent/US20210343150A1

## CPC

- G06V 20/584: {of vehicle lights or traffic lights}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G08G 1/0129: {for creating historical data or processing based on historical data}
- G08G 1/0145: {for active traffic flow control}
- G08G 1/04: using optical or ultrasonic detectors
- G08G 1/095: Traffic lights
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G08G 1/167: {Driving aids for lane monitoring, lane changing, e.g. blind spot detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

Methods and system are provided for training and using a model to determine states of lanes of interest. For instance, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light are received and used to train the model such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest. This model is then used by a vehicle in order to determine a state of a lane of interest. This state is then used to control the vehicle in an autonomous driving mode based on the state of the lane of interest.

## Claims

### Claim 1

A method of training a model for determining states of lanes of interest, the method comprising:
receiving, by one or more server computing devices, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light, the associated label generated by projecting a three-dimensional location of the at least one traffic light into the image; and
training, by the one or more server computing devices, the model using the image data such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, prior to the training, generating a second image using the image and the projection, the second image highlighting an area of the projection in the image, and wherein training the model is further based on the second image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, determining the state by processing the image to identify a blob of color within an area of the projection.

### Claim 4 (depends on 1)

The method of claim 1, wherein the lane state identifies whether a vehicle in that lane is required to go, stop, or use caution.

### Claim 5 (depends on 1)

The method of claim 1, further comprising, training the model to identify stop lines in images relevant to the lane of interest.

### Claim 6 (depends on 5)

The method of claim 5, wherein the training further includes using a label identifying a location of a stop line in the image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the image data further includes a second image and a second associated label identifying the at least one traffic light, a second state of the at least one traffic light in the second image, the second state being different from the state of the at least one traffic light such that the training includes using images of the at least one traffic light captured at different times in different states.

### Claim 8 (depends on 1)

The method of claim 1, further comprising, training the model to identify active and inactive elements of the at least one traffic light.

### Claim 9 (depends on 8)

The method of claim 8, wherein identifying includes whether the at least one traffic right is lit or unlit.

### Claim 10 (depends on 8)

The method of claim 8, wherein identifying includes determining a number of elements in the at least one traffic light.

### Claim 11 (depends on 1)

The method of claim 1, further comprising, training the model using images of the at least one traffic light captured during a period of time.

### Claim 12 (depends on 11)

The method of claim 11, wherein the model is a recurrent neural network or a long short term memory neural network.

### Claim 13 (depends on 11)

The method of claim 11, wherein the state of the at least one traffic light is determined by heuristics about traffic light patterns.

### Claim 14 (depends on 1)

The method of claim 1, wherein identifying the lane controlled by the at least one traffic light is achieved by projecting location of the lane controlled by the labeled at least one traffic light into the image.
