# US 2018/0322783 A1 · Systems and methods for visualizing potential risks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322783 A1 - Document ID: US20180322783A1 - Status: Published application - Kind: A1 - Application number: 15/625,595 - Filed: 2017-06-16 - Published: 2018-11-08 - Earliest claimed: 2017-05-05 - Assignees: Toyota Motor Engineering & Manufacturing North America, Inc.; Toyota Jidosha Kabushiki Kaisha - Applicants: Toyota Motor Engineering & Manufacturing North America, Inc.; Toyota Jidosha Kabushiki Kaisha - Inventors: Heishiro Toyoda; Hiroto Hamada - Web page: https://patentgrep.com/patent/US20180322783A1 ## CPC - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60K 35/21: using visual output, e.g. blinking lights or matrix displays - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60W 30/08: {Active safety systems} predicting or avoiding probable or impending collision {or attempting to minimise its consequences} - B60W 40/04: Traffic conditions - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60W 60/00276: {for two or more other traffic participants} - B62D 15/029: {Steering assistants using warnings or proposing actions to the driver without influencing the steering system} - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate - G06N 20/00: Machine learning - G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G07C 5/0825: {using optical means} - G07C 5/0841: {Registering performance data (recording measured values G01D; information storage G11B)} - G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]} - G08G 1/0962: having an indicator mounted inside the vehicle, e.g. giving voice messages - G08G 1/16: Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08) - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60K 35/25: using haptic output - B60K 35/265: {Voice} - B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions - B60K 2360/186: Displaying information according to relevancy - B60K 2360/334: Projection means - B60W 2554/4046: Behavior, e.g. aggressive or erratic - G02B 2027/014: {comprising information/image processing systems} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract System, methods, and other embodiments described herein relate to inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle. In one embodiment, a method includes identifying one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle. The method includes rendering, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment. ## Claims ### Claim 1 An engagement system for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle, comprising: one or more processors; a memory communicably coupled to the one or more processors and storing: a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to identify one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle, wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment forming circumstances that define the contextual hazards; and an engagement module including instructions that when executed by the one or more processors cause the one or more processors to render, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment, wherein the engagement module includes instructions to render the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object. ### Claim 2 (depends on 1) The engagement system of claim 1, wherein the potential hazards are likelihoods of risk defined according to detected objects and the circumstances in the surrounding environment, wherein the monitoring module includes instructions to identify the one or more potential hazards by analyzing the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment, and wherein the engagement module further includes instructions to render the one or more graphical elements by generating the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle. ### Claim 3 (depends on 2) The engagement system of claim 2, wherein the monitoring module includes instructions to identify the one or more potential hazards by analyzing the sensor data using at least the hazard model that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model. ### Claim 4 (depends on 2) The engagement system of claim 2, wherein the potential hazards are comprised of the contextual hazards, and discrete hazards, and wherein the discrete hazards are openly visible hazards that relate to detected objects that are on an intercepting trajectory with the vehicle. ### Claim 5 (depends on 1) The engagement system of claim 1, wherein the monitoring module further includes instructions to collect the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, and wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver. ### Claim 6 (depends on 1) The engagement system of claim 1, wherein the engagement module further includes the instructions to render the display scenario by animating the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a portrayal of the one or more potential hazards presently occurring, wherein the engagement module includes instructions to animate the graphical elements as an overlay on the surrounding environment using an AR display of the AR system within the vehicle. ### Claim 7 (depends on 6) The engagement system of claim 6, wherein the engagement module further includes the instructions to render the display scenario by rendering: animations of objects intersecting with a trajectory of the vehicle from unexpected locations, animations of the objects moving from behind blind corners, animations of the objects within zones proximate to the vehicle, animations of one of the objects moving along a diverging path of movement that is contrary to a current path of movement, and animations that highlight the zones within the surrounding environment associated with the potential hazards. ### Claim 8 (depends on 1) The engagement system of claim 1, wherein the AR system includes at least one display that is integrated with a windshield of the vehicle. ### Claim 9 A non-transitory computer-readable medium for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle and storing instructions that when executed by one or more processors cause the one or more processors to: identify one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle, wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment that form circumstances defining the contextual hazards; and render, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment, wherein the instructions to render include instructions to render the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the potential hazards are defined according to likelihoods of risk associated with detected objects and the circumstances in the surrounding environment, wherein the instructions to identify the one or more potential hazards include instructions to analyze the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the instructions to identify the one or more potential hazards include instructions to analyze the sensor data using the hazard model and a machine learning algorithm that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model, and wherein the instructions to render the one or more graphical elements include instructions to generate the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle. ### Claim 12 (depends on 9) The non-transitory computer-readable medium of claim 9, further comprising instructions to collect the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver. ### Claim 13 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the instructions to render the display scenario include instructions to animate the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a portrayal of the one or more potential hazards as presently occurring, wherein the instructions to the animate the graphical elements include instructions to animate the graphical elements as an overlay on the surrounding environment using an AR display of the AR system within the vehicle. ### Claim 14 A method for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle, comprising: identifying one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle, wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment forming circumstances that define the contextual hazards; and rendering, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment, wherein rendering the display scenario includes rendering the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object. ### Claim 15 (depends on 14) The method of claim 14, wherein the potential hazards are defined according to likelihoods of risk associated with detected objects and the circumstances in the surrounding environment, wherein identifying the one or more potential hazards includes analyzing the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment. ### Claim 16 (depends on 15) The method of claim 15, wherein identifying the one or more potential hazards includes analyzing the sensor data using the hazard model and a machine learning algorithm that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model, and wherein rendering the one or more graphical elements includes generating the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle. ### Claim 17 (depends on 15) The method of claim 15, wherein the potential hazards are comprised of the contextual hazards, and discrete hazards, and wherein the discrete hazards are openly visible hazards that relate to detected objects that are on intercepting trajectories with the vehicle. ### Claim 18 (depends on 14) The method of claim 14, further comprising: collecting the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver. ### Claim 19 (depends on 14) The method of claim 14, wherein rendering the display scenario includes animating the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a forecasted portrayal of the one or more potential hazards presently occurring, wherein the animating overlays the graphical elements on the surrounding environment using an AR display within the vehicle. ### Claim 20 (depends on 19) The method of claim 19, wherein rendering the display scenario includes rendering animations of objects intersecting with a projected trajectory of the vehicle from unexpected locations, animations of objects projected to move from behind blind corners, animations of objects within zones proximate to the vehicle, and animations that highlight the zones within the surrounding environment associated with the potential hazards.