# US 12,112,550 B2 · Systems and methods for traffic light detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,550 B2 - Document ID: US12112550B2 - Status: Granted - Kind: B2 - Application number: 17/129,661 - Filed: 2020-12-21 - Published: 2021-07-08 - Granted: 2024-10-08 - Earliest claimed: 2020-01-07 - Assignees: Motional AD LLC - Applicants: Motional Ad LLC - Inventors: Lubing Zhou; Jiong Yang; Ankit Dhall - Web page: https://patentgrep.com/patent/US12112550B2 ## CPC - G06V 20/584: {of vehicle lights or traffic lights} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G08G 1/09623: {Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle (G08G1/0967 takes precedence)} - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/743: Bracketing, i.e. taking a series of images with varying exposure conditions - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This disclosure describes the use of optical sensors to detect and characterize the state of traffic lights to assist with the navigation of autonomous vehicles. In particular, a specific optical configuration is shown that includes both a fixed-exposure sensor and an auto-exposure sensor. Imagery from the two sensor types can be combined to more accurately characterize the state of traffic signals at any particular intersection. Systems and methods for analyzing only select regions of the imagery captured by the traffic light detection system are also described. ## Claims ### Claim 1 A method, comprising: capturing a first image using a sensor coupled to an autonomous vehicle; obtaining, using a processing circuit, a location of the autonomous vehicle at the time the first image was captured; obtaining, using the processing circuit, a location of a traffic light; selecting, using the processing circuit, a first portion of the first image that includes the traffic light based on the obtained location of the traffic light and the obtained location of the autonomous vehicle; determining an orientation of a plurality of traffic lights in the first portion of the first image based on a shape of light emanating from each of the plurality traffic lights; identifying the traffic light from the plurality of traffic lights based on the determined orientations; determining, using the processing circuit, a first state of the identified traffic light based on the first portion of the first image; capturing a second image including the traffic light using the sensor; selecting a second portion of the second image, wherein a first ratio of a height of the traffic light in the first image to a height of the first portion of the first image is the same as a second ratio of a height of the traffic light in the second image to a height of the second portion of the second image; determining, using the processing circuit, a second state of the identified traffic light based on the second portion of the second image; and operating, by a control circuit, the autonomous vehicle in accordance with the first state and the second state of the identified traffic light. ### Claim 2 (depends on 1) The method as recited in claim 1, further comprising prior to selecting the first portion of the first image, correlating the obtained location of the traffic light relative to the obtained location of the autonomous vehicle with a location in the first image. ### Claim 3 (depends on 1) The method as recited in claim 1, wherein selecting the first portion of the first image includes determining a size of the selected first portion of the first image based on a distance between the autonomous vehicle and the traffic light. ### Claim 4 (depends on 1) The method as recited in claim 1, further comprising selecting portions of subsequently captured images based on subsequent locations of the autonomous vehicle relative to the traffic light. ### Claim 5 (depends on 1) The method as recited in claim 1, further comprising: selecting a third portion of the first image that includes another traffic light; and determining a state of the other traffic light, wherein operating, by the control circuit, the autonomous vehicle is performed in accordance with the determined state of the identified traffic light and the determined state of the other traffic light. ### Claim 6 (depends on 5) The method as recited in claim 5, further comprising: determining whether the state of the identified traffic light matches the state of the other traffic light. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein determining the state of the identified traffic light includes determining a color of a light emitted by the identified traffic light. ### Claim 8 (depends on 7) The method as recited in claim 7, wherein the determined state of the identified traffic light is a yellow arrow. ### Claim 9 (depends on 8) The method as recited in claim 8, further comprising determining, using the processing circuit, a direction of the arrow by referencing a traffic light database that includes a direction of one or more arrows known to be displayed by the identified traffic light. ### Claim 10 (depends on 1) The method as recited in claim 1, further comprising determining a confidence of the determined state of the identified traffic light. ### Claim 11 (depends on 10) The method as recited in claim 10, wherein the confidence of the determined state of the identified traffic light is based at least in part upon an area of the first image occupied by the light emitted by the identified traffic light with respect to an expected area of the first image occupied by the light emitted by the identified traffic light. ### Claim 12 (depends on 1) The method as recited in claim 1, wherein the sensor is a first optical sensor and wherein the method further comprises: capturing a third image using a second optical sensor; and selecting, using the processing circuit, a third portion of the third image that includes the traffic light, wherein determining the state of the identified traffic light is further based on the selected third portion of the third image. ### Claim 13 (depends on 12) The method as recited in claim 12, wherein the first optical sensor is a fixed-exposure sensor and the second optical sensor is an auto-exposure sensor. ### Claim 14 (depends on 13) The method as recited in claim 13, wherein operating parameters of the first optical sensor have values configured for detecting an average amount of illumination emitted by a traffic light. ### Claim 15 (depends on 1) The method as recited in claim 1, wherein: the first image is captured at a first distance from the traffic; the second image is captured at a second distance from the traffic light, the second distance being shorter than the first distance; and the second portion is larger than the first portion. ### Claim 16 (depends on 1) The method as recited in claim 1, wherein a height of the selected first portion of the first image is greater than a width of the selected first portion of the first image. ### Claim 17 (depends on 1) The method as recited in claim 1, wherein a size of the selected first portion of the first image is based on an estimated accuracy of a predicted location of the traffic light within the first image, wherein the estimated accuracy is based on historical data generated from previously captured images. ### Claim 18 (depends on 17) The method as recited in claim 17, wherein the height of the selected first portion of the first image is greater than the width of the selected first portion of the first image when an estimated accuracy of a vertical location of the traffic light in the first image is lower than an estimated accuracy of a horizontal location of the traffic light in the first image. ### Claim 19 A non-transitory computer-readable storage medium storing instructions configured to be executed by one or more circuits of an autonomous vehicle that cause the autonomous vehicle to carry out steps that include: capturing a first image using a sensor coupled to an autonomous vehicle; obtaining, using a processing circuit, a location of the autonomous vehicle at the time the first image was captured; obtaining, using the processing circuit, a location of a traffic light; selecting, using the processing circuit, a first portion of the first image that includes the traffic light based on the obtained location of the traffic light and the obtained location of the autonomous vehicle; determining an orientation of a plurality of traffic lights in the first portion of the first image based on a shape of light emanating from each of the plurality traffic lights; identifying the traffic light from the plurality of traffic lights based on the determined orientations; determining, using the processing circuit, a first state of the identified traffic light based on the first portion of the first image; capturing a second image including the traffic light using the sensor; selecting a second portion of the second image, wherein a first ratio of a height of the traffic light in the first image to a height of the first portion of the first image is the same as a second ratio of a height of the traffic light in the second image to a height of the second portion of the second image; determining, using the processing circuit, a second state of the identified traffic light based on the second portion of the second image; and operating, by a control circuit, the autonomous vehicle in accordance with the first state and the second state of the identified traffic light. ### Claim 20 An autonomous vehicle, comprising: a sensor; a processing circuit; a control circuit; and memory storing one or more programs configured to be executed by the circuits of the autonomous vehicle that cause the autonomous vehicle to carry out steps that include: capturing a first image using the sensor; obtaining, using the processing circuit, a location of the autonomous vehicle at the time the first image was captured; obtaining, using the processing circuit, a location of a traffic light; selecting, using the processing circuit, a first portion of the first image that includes the traffic light based on the obtained location of the traffic light and the obtained location of the autonomous vehicle; determining an orientation of a plurality of traffic lights in the first portion of the first image based on a shape of light emanating from each of the plurality traffic lights; identifying the traffic light from the plurality of traffic lights based on the determined orientations; determining, using the processing circuit, a first state of the identified traffic light based on the first portion of the first image; capturing a second image including the traffic light using the sensor; selecting a second portion of the second image, wherein a first ratio of a height of the traffic light in the first image to a height of the first portion of the first image is the same as a second ratio of a height of the traffic light in the second image to a height of the second portion of the second image; determining, using the processing circuit, a second state of the identified traffic light based on the second portion of the second image; and operating, by the control circuit, the autonomous vehicle in accordance with the first state and the second state of the identified traffic light.