# US 12,190,722 B2 · Method and system for traffic signal control with a learned model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,190,722 B2 - Document ID: US12190722B2 - Status: Granted - Kind: B2 - Application number: 17/838,772 - Filed: 2022-06-13 - Published: 2022-12-15 - Granted: 2025-01-07 - Earliest claimed: 2021-06-14 - Assignees: The Governing Council of the University of Toronto - Applicants: The Governing Council of the University of Toronto - Inventors: Parth Jaggi; Scott Sanner; Baher Abdulhai - Web page: https://patentgrep.com/patent/US12190722B2 ## CPC - G08G 1/0116: {from roadside infrastructure, e.g. beacons} - G08G 1/08: according to detected number or speed of vehicles - G06N 3/092: Reinforcement learning - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G08G 1/0129: {for creating historical data or processing based on historical data} - G08G 1/0145: {for active traffic flow control} - G08G 1/04: using optical or ultrasonic detectors - G08G 1/052: with provision for determining speed or overspeed {(speed measuring in general G01P)} - G08G 1/082: Controlling the time between beginning of the same phase of a cycle at adjacent intersections - G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]} - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract There is provided a system and method for traffic signal control of a traffic network with a learned model. The method including: receiving sensor readings from the traffic network, the sensor readings including positions and speeds of vehicles approaching each intersection; using a learned dynamics model that takes the sensor readings as input, predicting a plurality of possibilities for position and velocity of the vehicles approaching each intersection in a future timestep; determining a action for the one or more intersections by performing a tree search on the plurality of possibilities and selecting the possibility with a highest action value; and outputting the action to the traffic network for implementation as a traffic control action at the one or more intersections. ## Claims ### Claim 1 A method for traffic signal control of a traffic network with a learned model, the traffic network comprising one or more intersections and sensors associated with the intersections to determine vehicle traffic approaching each intersection, the method comprising, for each timestep: receiving sensor readings from the traffic network, the sensor readings comprising positions and speeds of vehicles approaching each intersection; using a learned dynamics model that takes the sensor readings as input, predicting a plurality of possibilities for position and velocity of the vehicles approaching each intersection in a future timestep; determining an action for the one or more intersections by performing a tree search on the plurality of possibilities and selecting the possibility with a highest action value; and outputting the action to the traffic network for implementation as a traffic control action at the one or more intersections. ### Claim 2 (depends on 1) The method of claim 1, wherein the action comprises a traffic light action that comprises either an extend action or a change action, wherein the extend action extends a current phase and the change action changes the current phase to a next phase in a predefined phase cycle of a traffic light of the intersection. ### Claim 3 (depends on 2) The method of claim 2, wherein the dynamics model is trained using a simulation of traffic at the one or more intersections, the simulation artificially generates traffic demand that varies for each approach of each of the one or more intersections, where the change action is randomly selected. ### Claim 4 (depends on 1) The method of claim 1, wherein the dynamics model is trained using vehicle movement data collected from one or more intersections in real-life. ### Claim 5 (depends on 1) The method of claim 1, wherein the reinforcement model comprises a vehicle-level model that, for each vehicle, takes as input the position and the velocity of the vehicle on an associated lane, the position and velocity of a downstream vehicle, a phase history, and a current action, to predict the position and the velocity of the vehicle at the future timestep. ### Claim 6 (depends on 5) The method of claim 5, wherein the phase history comprises a timestep history of phases corresponding to the associated lane. ### Claim 7 (depends on 1) The method of claim 1, wherein the tree search comprises performing a Monte-Carlo Tree Search (MCTS). ### Claim 8 (depends on 7) The method of claim 7, wherein the MCTS comprises: using a tree policy, traversing a tree path to reach a leaf node; performing expansion where the leaf node is non-terminal; expanding the tree by adding child nodes to the leaf node; performing simulation using a rollout policy to simulate a trajectory up to a predetermined condition; and performing backup by updating action values of the traversed path inside the tree using the performed simulation. ### Claim 9 (depends on 8) The method of claim 8, wherein the tree policy comprises performing upper confidence bound selection. ### Claim 10 (depends on 8) The method of claim 8, wherein the rollout policy selects a random timestep between a minimum and a maximum green-light time to perform the change action, and repeats the selection until a single phase cycle is completed and rewards obtained over this cycle is used to estimate an initial state value. ### Claim 11 A system for traffic signal control of a traffic network with a learned model of a traffic network, the traffic network comprising one or more intersections and sensors associated with the intersections to determine vehicle traffic approaching each intersection, the system comprising one or more processors and a data storage, the one or more processors configurable to execute at each timestep: an input module to receive sensor readings from the traffic network, the sensor readings comprising positions and speeds of vehicles approaching each intersection; a machine learning module to, using a dynamics model that takes the sensor readings as input, predict a plurality of possibilities for position and velocity of the vehicles approaching each intersection in a future timestep; a selection module to determine an action for the one or more intersections by performing a tree search on the plurality of possibilities and selecting the possibility with a highest action value; and an action module to output the action to the traffic network for implementation as a traffic control action at the one or more intersections. ### Claim 12 (depends on 11) The system of claim 11, wherein the action comprises a traffic light action that comprises either an extend action or a change action, wherein the extend action extends a current phase and the change action changes the current phase to a next phase in a predefined phase cycle of a traffic light of the intersection. ### Claim 13 (depends on 12) The system of claim 12, wherein the dynamics model is trained using a simulation of traffic at the one or more intersections, the simulation artificially generates traffic demand that varies for each approach of each of the one or more intersections, where the change action is randomly selected. ### Claim 14 (depends on 11) The system of claim 11, wherein the dynamics model is trained using vehicle movement data collected from one or more intersections in real-life. ### Claim 15 (depends on 11) The system of claim 11, wherein the reinforcement model comprises a vehicle-level model that, for each vehicle, takes as input the position and the velocity of the vehicle on an associated lane, the position and velocity of a downstream vehicle, a phase history, and a current action, to predict the position and the velocity of the vehicle at the future timestep. ### Claim 16 (depends on 15) The system of claim 15, wherein the phase history comprises a timestep history of phases corresponding to the associated lane. ### Claim 17 (depends on 11) The system of claim 11, wherein the tree search comprises performing a Monte-Carlo Tree Search (MCTS). ### Claim 18 (depends on 17) The system of claim 17, wherein the MCTS comprises: using a tree policy, traversing a tree path to reach a leaf node; performing expansion where the leaf node is non-terminal; expanding the tree by adding child nodes to the leaf node; performing simulation using a rollout policy to simulate a trajectory up to a predetermined condition; and performing backup by updating action values of the traversed path inside the tree using the performed simulation. ### Claim 19 (depends on 18) The system of claim 18, wherein the tree policy comprises performing upper confidence bound selection. ### Claim 20 (depends on 18) The system of claim 18, wherein the rollout policy selects a random timestep between a minimum and a maximum green-light time to perform the change action, and repeats the selection until a single phase cycle is completed and rewards obtained over this cycle is used to estimate an initial state value.