US 10,467,487 B1Granted

Fusion-based traffic light recognition for autonomous driving

AssigneesChongqing Jinkang New Energy Automobile Co., Ltd., SF Motors, Inc.

InventorsFan Wang, Huajun Chai, Ke Huang, Saurabh Singh, Rui Yu, Yifan Tang

Life of this patent

Estimating the term…

Loading patent…

Abstract

Various arrangements are presented for determining a state of a traffic light. A first analysis of a scene may be performed that includes the traffic light. The first analysis may result in the output of a first classification result and a first confidence score. A second, different analysis of the scene may be performed that also outputs a classification result and a confidence score. A fusion process may be performed to fuse the classification results into a fused classification result based on the confidence scores and one or more weighting metrics. A vehicle may then be driven at least partially based on the fused classification result.

Claims18

Dependent claims

A method for determining a state of a traffic light, the method comprising:

performing, by an onboard vehicle processing system, a first analysis of a scene that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality of probabilities corresponds to a different traffic light state;
performing, by the onboard vehicle processing system, a second analysis of the scene that includes the traffic light, wherein:
the second analysis is a different type of analysis than the first analysis;
the second analysis outputs a second plurality of probabilities;
and
each probability of the second plurality of probabilities corresponds to a different traffic light state;
performing, by the onboard vehicle processing system, a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of:
an offline weighting metric;
and an online weighting metric;
determining the online weighting metric, which comprises:
determining that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;
in response to determining the first traffic light state is different than the second traffic light state, performing a traffic light sequence analysis to determine an expected classification result;
and
comparing the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state;
and
driving, by the onboard vehicle processing system, a vehicle at least partially based on the fused classification result.

13 more elements

A system for determining a state of a traffic light, the system comprising:

one or more image sensors attached with a vehicle;
an onboard vehicle processing system, comprising one or more processors, that receives images from the one or more image sensors, wherein the onboard vehicle processing system is located onboard the vehicle, wherein the one or more processors are configured to:
perform a first analysis of a scene from a first image received from the one or more image sensors that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state;
perform a second analysis of the scene from a second image received from the one or more image sensors that includes the traffic light, wherein:
the second analysis is a different type of analysis than the first analysis;
the second analysis outputs a second plurality of probabilities;
and
each probability of the second plurality corresponds to a different traffic light state;
perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of:
an offline weighting metric;
and an online weighting metric, wherein:
the online weighting metric is determined by the one or more processors of the onboard vehicle processing system being configured to:
determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;
perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state;
and
compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state;
and
drive the vehicle at least partially based on the fused classification result.

15 more elements

A non-transitory processor-readable medium for determining the state of a traffic light comprising processor-readable instructions configured to cause one or more processors to:

perform a first analysis of a scene from a first image that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state;
perform a second analysis of the scene from a second image that includes the traffic light, wherein:
the second analysis is a different type of analysis than the first analysis;
the second analysis outputs a second plurality of probabilities;
and
each probability of the second plurality corresponds to a different traffic light state;
perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of:
an offline weighting metric;
and an online weighting metric, wherein:
the online weighting metric is determined by the processor-readable instructions being configured to cause the one or more processors to:
determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;
perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state;
and
compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state;
and
cause a vehicle to be driven at least partially based on the fused classification result.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/216,731Filed 11 Dec 2018
Granted: 5 Nov 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06V 20/584
Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or…
B60R 1/005
Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or…:…
G05D 1/00
Control of position, course, altitude or attitude of land, water, air or
G05D 1/0231
Specially adapted to land vehicles: using optical position detecting means