# US 2022/0036732 A1 · Bulb-wise traffic light state, signal, and transition estimator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036732 A1 - Document ID: US20220036732A1 - Status: Published application - Kind: A1 - Application number: 16/984,006 - Filed: 2020-08-03 - Published: 2022-02-03 - Earliest claimed: 2020-08-03 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Kun-Hsin Chen; Hai Jin; Shunsho Kaku; Takaaki Tagawa; Ryan W. Wolcott; Hiroshi Yasuda - Web page: https://patentgrep.com/patent/US20220036732A1 ## CPC - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G08G 1/09623: {Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle (G08G1/0967 takes precedence)} - B60W 60/0016: {of the vehicle or its occupants} - G01C 21/3811: {Point data, e.g. Point of Interest [POI]} - G06T 7/60: Analysis of geometric attributes - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 10/56: relating to colour - G06V 10/82: using neural networks - G06V 20/584: {of vehicle lights or traffic lights} - G08G 1/096725: {where the received information generates an automatic action on the vehicle control} - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G08G 1/096827: {where the route is computed onboard} - G08G 1/096877: {where the input to the navigation device is provided by a suitable I/O arrangement (details of I/O arrangements for onboard navigation computers G01C21/36; I/O arrangements for general purpose computers G06F3/00)} - G08G 1/0969: having a display in the form of a map - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods of traffic light detection/perception are provided for detecting traffic lights states and/or transitions for different traffic light layouts. Basic traffic light information can be obtained regarding, e.g., the existence of a traffic light(s) at an intersection being approached by a vehicle, and which bulb(s) may apply to/control a particular lane of traffic at the intersection. Based on that information, bulb-specific or bulbwise detection can be performed. Upon determining the existence of a traffic light, specific bulbs within or making up the traffic light can be detected, and their various states can be predicted or estimated relative to a corresponding bulb group. The various states can then be simplified and output as a message(s) upon which autonomous control of a vehicle can be based. ## Claims ### Claim 1 A method, comprising: generating images of a traffic light; detecting current characteristics of each bulb of a plurality of bulbs comprising the traffic light based on the images; temporally filtering the current characteristics of each bulb in one or more bulb groups to refine the detected current characteristics; and outputting one or more bulb-specific states regarding the one or more bulb groups. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating an autonomous vehicle navigation plan based on the one or more bulb-specific states. ### Claim 3 (depends on 1) The method of claim 1, wherein the generating of the images comprises projecting information regarding the traffic light from a high definition (HD) map onto a plurality of in-vehicle camera-captured images of the traffic light. ### Claim 4 (depends on 3) The method of claim 3, wherein the one or more bulb groups are defined in the HD map. ### Claim 5 (depends on 4) The method of claim 4, wherein the one or more bulb groups are defined relative to a lane or lanes of roadway controlled by the traffic light. ### Claim 6 (depends on 5) The method of claim 5, wherein the detecting of the current characteristics comprises determining a data intersection between each bulb of each of the one or more bulb groups and characteristics identified via the in-vehicle camera-captured images. ### Claim 7 (depends on 6) The method of claim 6, further comprising applying an optional bulb-specific filter refining accuracy of the current characteristics. ### Claim 8 (depends on 3) The method of claim 3, wherein the detecting of the current characteristics further comprises detecting at least one of a bounding box location, a color, a geometry, a flashing state, an angular direction, and an activation state. ### Claim 9 (depends on 1) The method of claim 1, wherein the temporally filtering of the current characteristics comprises applying a bulb group filter to filter out erroneously detected characteristics. ### Claim 10 (depends on 9) The method of claim 9, wherein the bulb group filter is designed based on regional traffic light bulb color transitions. ### Claim 11 (depends on 1) The method of claim 1, wherein the outputting of the one or more bulb-specific states comprises a simplified bulb state upon which one or more autonomous control signals is based. ### Claim 12 (depends on 11) The method of claim 11, wherein the simplified bulb state comprises a go-no-go state for each of the one or more bulb groups. ### Claim 13 A vehicle comprising: a camera; a bulb-specific detector, comprising: a bulb detector component detecting current characteristics of each bulb of a plurality of bulbs comprising the traffic light based on images captured by the camera; a post processor translating the current characteristics of each bulb into bulb group-based characteristics; and a temporal filter filtering out erroneous predictions of the current characteristics; and an autonomous control system receiving bulb-specific output messages from the bulb-specific detector upon which autonomous control signals for controlling the vehicle are based. ### Claim 14 (depends on 13) The vehicle of claim 13, further comprising a map database storing map information including location information regarding the traffic light and bulb group information regarding the traffic light. ### Claim 15 (depends on 14) The vehicle of claim 14, wherein the map information regarding the traffic light and bulb group information is overlaid onto the images captured by the camera to create image thumbnails for analysis by the bulb detector component for the detecting of the current characteristics of each bulb. ### Claim 16 (depends on 14) The vehicle of claim 14, wherein the post processor matches the current characteristics to each bulb of one or more bulb groups based on the bulb group information. ### Claim 17 (depends on 13) The vehicle of claim 13, wherein the temporal filter filters out erroneous bulb-group-specific characteristics in accordance with known traffic light bulb translations particular to a region bring traveled by the vehicle. ### Claim 18 (depends on 13) The vehicle of claim 13, wherein the bulb-specific detector further comprises a bulbwise filter filtering our erroneous detected characteristics of each bulb. ### Claim 19 (depends on 13) The vehicle of claim 13, wherein the bulb-specific detector outputs simplified bulb-group-specific characteristics to the autonomous control system. ### Claim 20 (depends on 13) The vehicle of claim 13, wherein the current characteristics comprise at least one of a bounding box location, a color, a geometry, a flashing state, an angular direction, and an activation state.