# US 11,442,451 B2 · Systems and methods for generating driving recommendations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,451 B2 - Document ID: US11442451B2 - Status: Granted - Kind: B2 - Application number: 16/833,774 - Filed: 2020-03-30 - Published: 2021-09-30 - Granted: 2022-09-13 - Earliest claimed: 2020-03-30 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Stephen G. McGill, Jr.; Guy Rosman; Luke S. Fletcher; Brent Schlotfeldt - Web page: https://patentgrep.com/patent/US11442451B2 ## CPC - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G08G 1/167: {Driving aids for lane monitoring, lane changing, e.g. blind spot detection} - G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)} - G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 5/02: Knowledge representation; Symbolic representation - G08G 1/0108: {based on the source of data} - G08G 1/0129: {for creating historical data or processing based on historical data} - G08G 1/0145: {for active traffic flow control} - G08G 1/096708: {where the received information might be used to generate an automatic action on the vehicle control} - G08G 1/096791: {where the origin of the information is another vehicle} - G08G 1/163: {involving continuous checking} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for generating driving recommendations are disclosed herein. One embodiment divides automatically a roadway into a plurality of lane-level cells; generates a graph network that represents the plurality of lane-level cells; gathers information pertaining to one or more detected road agents; projects onto the graph network the gathered information pertaining to the one or more detected road agents to update a current status of the plurality of lane-level cells; processes the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells, the predicted future status including at least occupancy, by a detected road agent, of the respective lane-level cells in the plurality of lane-level cells; and generates a driving recommendation based, at least in part, on the predicted future status of the plurality of lane-level cells. ## Claims ### Claim 1 A system for generating driving recommendations, the system comprising: one or more processors; and a memory communicably coupled to the one or more processors and storing: a graph-construction module including instructions that when executed by the one or more processors cause the one or more processors to: divide automatically a roadway into a plurality of lane-level cells; and generate a graph network that represents the plurality of lane-level cells; a graph-update module including instructions that when executed by the one or more processors cause the one or more processors to: gather information pertaining to one or more detected road agents; and project onto the graph network the gathered information pertaining to the one or more detected road agents to update a current status of the plurality of lane-level cells; a prediction module including instructions that when executed by the one or more processors cause the one or more processors to process the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells, the predicted future status including at least occupancy, by a detected road agent, of the respective lane-level cells in the plurality of lane-level cells; and a recommendation module including instructions that when executed by the one or more processors cause the one or more processors to generate a driving recommendation based, at least in part, on the predicted future status of the plurality of lane-level cells. ### Claim 2 (depends on 1) The system of claim 1, wherein each lane-level cell in the plurality of lane-level cells is less than 2 meters in length in a longitudinal direction of the roadway. ### Claim 3 (depends on 1) The system of claim 1, wherein the gathered information pertaining to the one or more detected road agents includes one or more of position data, velocity data, raw sensor data, object-detection data, image-segmentation data, data pertaining to view obstructions and occlusions, map data, time of day, day of week, weather data, and temperature data. ### Claim 4 (depends on 1) The system of claim 1, wherein the graph network includes a plurality of nodes and the instructions in the prediction model to process the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells include instructions to run one of a discrete Gaussian kernel and a custom asymmetrical kernel via convolution over the plurality of nodes. ### Claim 5 (depends on 1) The system of claim 1, wherein the instructions in the prediction model to process the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells include instructions to: encode the graph network using a graph neural network (GNN) and a variational autoencoder (VAE) to produce an encoded graph network; process the encoded graph network using a long short-term memory (LSTM) network to predict the future status of the plurality of lane-level cells; and decode the encoded graph network using the GNN to produce a decoded graph network. ### Claim 6 (depends on 1) The system of claim 1, wherein the system is deployed in an ego vehicle and the driving recommendation includes a risk estimate pertaining to a future maneuver by the ego vehicle. ### Claim 7 (depends on 6) The system of claim 6, wherein the ego vehicle is an autonomous vehicle. ### Claim 8 (depends on 6) The system of claim 6, wherein the recommendation module includes further instructions to communicate the risk estimate to an occupant of the ego vehicle. ### Claim 9 (depends on 1) The system of claim 1, wherein the system is deployed in an ego vehicle, the driving recommendation includes a risk estimate pertaining to behavior of a particular one of the one or more detected road agents, and the recommendation module includes further instructions to communicate the risk estimate to an occupant of the ego vehicle. ### Claim 10 (depends on 1) The system of claim 1, wherein the gathered information pertaining to the one or more detected road agents projected onto the graph network includes occupancy, velocity, and emission probability, wherein the emission probability is a probability that a given detected road agent will exit a current lane-level cell in the plurality of lane-level cells to enter a different lane-level cell in the plurality of lane-level cells within a predetermined time horizon. ### Claim 11 A non-transitory computer-readable medium for generating driving recommendations and storing instructions that when executed by one or more processors cause the one or more processors to: divide automatically a roadway into a plurality of lane-level cells; generate a graph network that represents the plurality of lane-level cells; gather information pertaining to one or more detected road agents; project onto the graph network the gathered information pertaining to the one or more detected road agents to update a current status of the plurality of lane-level cells; process the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells, the predicted future status including at least occupancy, by a detected road agent, of the respective lane-level cells in the plurality of lane-level cells; and generate a driving recommendation based, at least in part, on the predicted future status of the plurality of lane-level cells. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the driving recommendation includes a risk estimate pertaining to a future maneuver by an ego vehicle. ### Claim 13 A method of generating driving recommendations, the method comprising: dividing automatically a roadway into a plurality of lane-level cells; generating a graph network that represents the plurality of lane-level cells; gathering information pertaining to one or more detected road agents; projecting onto the graph network the gathered information pertaining to the one or more detected road agents to update a current status of the plurality of lane-level cells; processing the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells, the predicted future status including at least occupancy, by a detected road agent, of the respective lane-level cells in the plurality of lane-level cells; and generating a driving recommendation based, at least in part, on the predicted future status of the plurality of lane-level cells. ### Claim 14 (depends on 13) The method of claim 13, wherein each lane-level cell in the plurality of lane-level cells is less than 2 meters in length in a longitudinal direction of the roadway. ### Claim 15 (depends on 13) The method of claim 13, wherein the gathered information pertaining to the one or more detected road agents includes one or more of position data, velocity data, raw sensor data, object-detection data, image-segmentation data, data pertaining to view obstructions and occlusions, map data, time of day, day of week, weather data, and temperature data. ### Claim 16 (depends on 13) The method of claim 13, wherein the graph network includes a plurality of nodes and processing the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells includes running one of a discrete Gaussian kernel and a custom asymmetrical kernel via convolution over the plurality of nodes. ### Claim 17 (depends on 13) The method of claim 13, wherein processing the graph network based on the updated current status of the plurality of lane-level cells to predict a future status of the plurality of lane-level cells includes: encoding the graph network using a graph neural network (GNN) and a variational autoencoder (VAE) to produce an encoded graph network; processing the encoded graph network using a long short-term memory (LSTM) network to predict the future status of the plurality of lane-level cells; and decoding the encoded graph network using the GNN to produce a decoded graph network. ### Claim 18 (depends on 13) The method of claim 13, wherein the driving recommendation includes a risk estimate pertaining to a future maneuver by an ego vehicle. ### Claim 19 (depends on 18) The method of claim 18, further comprising communicating the risk estimate to an occupant of the ego vehicle. ### Claim 20 (depends on 13) The method of claim 13, wherein the gathered information pertaining to the one or more detected road agents projected onto the graph network includes occupancy, velocity, and emission probability, wherein the emission probability is a probability that a given detected road agent will exit a current lane-level cell in the plurality of lane-level cells to enter a different lane-level cell in the plurality of lane-level cells within a predetermined time horizon.