# US 12,592,171 B2 · Vehicular driving assist system with head up display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,592,171 B2 - Document ID: US12592171B2 - Status: Granted - Kind: B2 - Application number: 18/480,686 - Filed: 2023-10-04 - Published: 2024-04-11 - Granted: 2026-03-31 - Earliest claimed: 2022-10-07 - Assignees: Magna Electronics Inc. - Applicants: Magna Electronics Inc. - Inventors: Özgür Nurettin Püskül - Web page: https://patentgrep.com/patent/US12592171B2 ## CPC - G09G 3/002: {to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT} - A63F 13/245: specially adapted to a particular type of game, e.g. steering wheels - A63F 13/25: Output arrangements for video game devices - A63F 13/803: Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks - A63F 13/90: Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets - B60R 1/22: for viewing an area outside the vehicle, e.g. the exterior of the vehicle - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - B60R 2300/205: using a head-up display - B60R 2300/308: by overlaying the real scene, e.g. through a head-up display on the windscreen - B60R 2300/8093: for obstacle warning - G06T 2207/30201: Face - G06T 2207/30241: Trajectory - G06T 2207/30261: Obstacle - G06T 2207/30268: Vehicle interior - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A vehicular driving assist system includes a plurality of sensors disposed at a vehicle that sense exterior of an equipped vehicle. The system includes a head-up display disposed at the equipped vehicle configured to display one or more visual elements in a field of view of a driver while the driver views forward through a windshield. The system, responsive to processing at an ECU of sensor data captured by the sensors, detects another vehicle present exterior the equipped vehicle. The vehicular driving assist system displays a plurality of visual elements via the head-up display. The plurality of visual elements includes (i) a bounding box appearing to the driver to surround the detected other vehicle when the detected other vehicle is viewable by the driver through the windshield of the equipped vehicle, (ii) a current speed of the detected other vehicle and (iii) a predicted trajectory of the detected other vehicle. ## Claims ### Claim 1 A vehicular driving assist system, the vehicular driving assist system comprising: a plurality of sensors disposed at a vehicle equipped with the vehicular driving assist system, the plurality of sensors sensing exterior of the equipped vehicle, wherein the plurality of sensors are operable to capture sensor data; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein sensor data captured by the plurality of sensors is transferred to and is processed at the ECU; a head-up display disposed at the equipped vehicle, wherein the head-up display is configured to display one or more visual elements in a field of view of a driver of the equipped vehicle while the driver views forward of the equipped vehicle through a windshield of the equipped vehicle; wherein the vehicular driving assist system, responsive to processing at the ECU of sensor data captured by the plurality of sensors and transferred to the ECU, detects another vehicle present exterior the equipped vehicle; and wherein the vehicular driving assist system, responsive at least in part to processing at the ECU of sensor data captured by the plurality of sensors, displays a plurality of visual elements via the head-up display, and wherein the plurality of visual elements comprises (i) a bounding box appearing to the driver to surround the detected other vehicle when the detected other vehicle is viewable by the driver through the windshield of the equipped vehicle, (ii) an information box appearing to visually overlay the driver's view of the detected other vehicle, wherein the information box comprises a current relative speed of the detected other vehicle relative to current speed of the equipped vehicle and (iii) a predicted trajectory of the detected other vehicle. ### Claim 2 (depends on 1) The vehicular driving assist system of claim 1, wherein a position of the plurality of visual elements on the head-up display is based on a position of the detected other vehicle relative to the equipped vehicle. ### Claim 3 (depends on 1) The vehicular driving assist system of claim 1, wherein the bounding box changes color based on location of the detected other vehicle relative to the equipped vehicle. ### Claim 4 (depends on 3) The vehicular driving assist system of claim 3, wherein the bounding box changes to a red color responsive to at least one selected from the group consisting of (i) the detected other vehicle approaching the equipped vehicle at a speed greater than a threshold speed, (ii) the detected other vehicle being within a threshold distance to the equipped vehicle, and (iii) the driver of the equipped vehicle commencing a change to an adjacent traffic lane when the detected other vehicle is present in the adjacent traffic lane. ### Claim 5 (depends on 1) The vehicular driving assist system of claim 1, wherein the plurality of sensors comprises a camera viewing exterior of the equipped vehicle, and wherein the camera comprises a CMOS imaging array having at least one million photosensors arranged in rows and columns. ### Claim 6 (depends on 5) The vehicular driving assist system of claim 5, wherein the camera is at least partially disposed in a space between a headliner of the equipped vehicle and the roof of the equipped vehicle. ### Claim 7 (depends on 5) The vehicular driving assist system of claim 5, further comprising a solar panel disposed at a roof of the equipped vehicle, and wherein the solar panel at least partially powers the camera. ### Claim 8 (depends on 1) The vehicular driving assist system of claim 1, wherein the plurality of sensors comprises at least one radar sensor sensing exterior of the equipped vehicle. ### Claim 9 (depends on 1) The vehicular driving assist system of claim 1, wherein the plurality of sensors comprises at least one lidar sensor sensing exterior of the equipped vehicle. ### Claim 10 (depends on 1) The vehicular driving assist system of claim 1, wherein the plurality of sensors comprises (i) at least one camera viewing exterior of the equipped vehicle and (ii) at least one radar sensor sensing exterior of the equipped vehicle. ### Claim 11 (depends on 1) The vehicular driving assist system of claim 1, wherein the plurality of visual elements represent a portion of an interactive game for the driver of the equipped vehicle. ### Claim 12 (depends on 11) The vehicular driving assist system of claim 11, wherein the vehicular driving assist system controls an aspect of the interactive game based on user input received from the driver of the equipped vehicle. ### Claim 13 (depends on 12) The vehicular driving assist system of claim 12, wherein, when driver-controlled steering of the equipped vehicle is electrically decoupled from controlling steering of the equipped vehicle, the user input comprises a steering wheel of the equipped vehicle. ### Claim 14 (depends on 1) The vehicular driving assist system of claim 1, further comprising an interior driver monitoring camera, the interior driver monitoring camera capturing image data, wherein image data captured by the interior driver monitoring camera is transferred to and processed at the ECU, and wherein the vehicular driving assist system, responsive to processing at the ECU of image data captured by the interior driver monitoring camera and transferred to the ECU, determines a location of the driver's eyes, and wherein the vehicular driving assist system displays the plurality of visual elements based at least in part on the determined location of the driver's eyes and a determined location of the detected other vehicle relative to the equipped vehicle. ### Claim 15 (depends on 1) The vehicular driving assist system of claim 1, wherein the vehicular driving assist system, responsive to a user input, simulates an advanced driver assist system (ADAS) function using the plurality of visual elements. ### Claim 16 (depends on 15) The vehicular driving assist system of claim 15, wherein the ADAS function comprises an adaptive cruise control function. ### Claim 17 A vehicular driving assist system, the vehicular driving assist system comprising: a plurality of sensors disposed at a vehicle equipped with the vehicular driving assist system, the plurality of sensors sensing exterior of the equipped vehicle, wherein the plurality of sensors are operable to capture sensor data, and wherein the plurality of sensors comprises (i) at least one camera viewing exterior of the equipped vehicle and (ii) at least one radar sensor sensing exterior of the equipped vehicle; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein sensor data captured by the plurality of sensors is transferred to and is processed at the ECU; a head-up display disposed at the equipped vehicle, wherein the head-up display is configured to display one or more visual elements in a field of view of a driver of the equipped vehicle while the driver views forward of the equipped vehicle through a windshield of the equipped vehicle; wherein the vehicular driving assist system, responsive to processing at the ECU of sensor data captured by the plurality of sensors and transferred to the ECU, detects another vehicle present exterior the equipped vehicle; wherein the vehicular driving assist system, responsive at least in part to processing at the ECU of sensor data captured by the plurality of sensors, displays a plurality of visual elements via the head-up display, and wherein the plurality of visual elements comprises (i) a bounding box appearing to the driver to surround the detected other vehicle when the detected other vehicle is viewable by the driver through the windshield of the equipped vehicle, (ii) an information box appearing to visually overlay the driver's view of the detected other vehicle, wherein the information box comprises a current relative speed of the detected other vehicle relative to current speed of the equipped vehicle and (iii) a predicted trajectory of the detected other vehicle; and wherein a position of the plurality of visual elements on the head-up display is based on a position of the detected other vehicle relative to the equipped vehicle. ### Claim 18 (depends on 17) The vehicular driving assist system of claim 17, wherein the bounding box changes color based on location of the detected other vehicle relative to the equipped vehicle. ### Claim 19 (depends on 17) The vehicular driving assist system of claim 17, wherein the plurality of sensors comprises at least one lidar sensor sensing exterior of the equipped vehicle. ### Claim 20 (depends on 17) The vehicular driving assist system of claim 17, wherein the vehicular driving assist system displays the plurality of visual elements based at least in part on the determined location of the driver's eyes. ### Claim 21 A vehicular driving assist system, the vehicular driving assist system comprising: a plurality of sensors disposed at a vehicle equipped with the vehicular driving assist system, the plurality of sensors sensing exterior of the equipped vehicle, wherein the plurality of sensors are operable to capture sensor data; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein sensor data captured by the plurality of sensors is transferred to and is processed at the ECU; a head-up display disposed at the equipped vehicle, wherein the head-up display is configured to display one or more visual elements in a field of view of a driver of the equipped vehicle while the driver views forward of the equipped vehicle through a windshield of the equipped vehicle; wherein the vehicular driving assist system, responsive to processing at the ECU of sensor data captured by the plurality of sensors and transferred to the ECU, detects another vehicle present exterior the equipped vehicle; wherein the vehicular driving assist system, responsive at least in part to processing at the ECU of sensor data captured by the plurality of sensors, displays a plurality of visual elements via the head-up display, and wherein the plurality of visual elements comprises (i) a bounding box appearing to the driver to surround the detected other vehicle when the detected other vehicle is viewable by the driver through the windshield of the equipped vehicle, (ii) an information box appearing to visually overlay the driver's view of the detected other vehicle, wherein the information box comprises a current relative speed of the detected other vehicle relative to current speed of the equipped vehicle and (iii) a predicted trajectory of the detected other vehicle; and wherein the plurality of visual elements further comprise a plurality of markers, and wherein the plurality of markers represent a portion of an interactive game for the driver of the equipped vehicle. ### Claim 22 (depends on 21) The vehicular driving assist system of claim 21, wherein the vehicular driving assist system controls an aspect of the interactive game based on user input received from the driver of the equipped vehicle. ### Claim 23 (depends on 22) The vehicular driving assist system of claim 22, wherein, when driver-controlled steering of the equipped vehicle is electrically decoupled from controlling steering of the equipped vehicle, the user input comprises a steering wheel of the equipped vehicle. ### Claim 24 (depends on 21) The vehicular driving assist system of claim 21, wherein the vehicular driving assist system displays the plurality of visual elements based at least in part on the determined location of the driver's eyes and a determined location of the detected other vehicle relative to the equipped vehicle.