# US 10,766,498 B2 · Method, apparatus, and computer readable storage medium having instructions for controlling a display of an augmented reality display device for a transportation vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,766,498 B2 - Document ID: US10766498B2 - Status: Granted - Kind: B2 - Application number: 16/123,221 - Filed: 2018-09-06 - Published: 2019-03-14 - Granted: 2020-09-08 - Earliest claimed: 2017-09-12 - Assignees: Volkswagen Aktiengesellschaft - Applicants: Volkswagen Aktiengesellschaft - Inventors: Andro Kleen; Daniel Morales Fernández; Adrian Benjamin Haeske; Adrian Haar; Vitalij Sadovitch - Web page: https://patentgrep.com/patent/US10766498B2 ## CPC - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01) - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60Q 9/008: {for anti-collision purposes} - G01C 21/3697: {Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence)} - G02B 27/01: Head-up displays - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G08G 1/0962: having an indicator mounted inside the vehicle, e.g. giving voice messages - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60K 2360/177: Augmented reality - B60K 2360/178: Warnings - B60R 2300/205: using a head-up display - B60R 2300/308: by overlaying the real scene, e.g. through a head-up display on the windscreen - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 2050/146: {Display means} - G02B 2027/0141: {characterised by the informative content of the display} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract A method, a device, and a computer-readable storage medium with instructions for controlling a display of an augmented reality display device for a transportation vehicle. The presence of a driving situation is detected, in which a warning is displayed to a driver of the transportation vehicle; the augmented reality display device generates a virtual object for display; the virtual object visualizes potential imminent events, actions or dangerous situations; the virtual object has moving graphical elements that simulate a movement of the virtual object; and the augmented reality display device outputs the generated virtual object for display. ## Claims ### Claim 1 A device for controlling a display of an augmented reality display device for a transportation vehicle, the device comprising: an analysis unit for detecting a driving situation in which a warning is to be displayed to a driver of the transportation vehicle; a graphics generator for generating virtual objects for display by the augmented reality display device, wherein the virtual objects have moving graphical elements which simulate movement of the virtual objects; and an output for outputting the virtual objects for display by the augmented reality display device to enable visualization of possible imminent events, actions or hazardous situations via the augmented reality display device, wherein the virtual objects displayed on the augmented reality display device include representations of actual objects in an environment surrounding the vehicle, virtual objects that indicate an increased risk of objects depicted by the virtual objects actually being present in the environment surrounding the transportation vehicle, and virtual objects providing instruction for the driver to drive the transportation vehicle to position the transportation vehicle relative to locations within the environment surrounding the transportation vehicle so as to assist the driver to follow transportation vehicle navigation instructions issued by a navigation system of the transportation vehicle or parking maneuvering assistance instructions generated by the transportation vehicle. ### Claim 2 (depends on 1) The device of claim 1, wherein the moving graphical elements form a group of light points that form a virtual object. ### Claim 3 (depends on 1) The device of claim 1, wherein at least one of the virtual objects represents an actual or potential hazardous object in the environment of the transportation vehicle. ### Claim 4 (depends on 3) The device of claim 3, wherein a position or a movement of the at least one virtual object representing the actual or potential hazardous object does not coincide with a position or movement of an actual hazardous object. ### Claim 5 (depends on 3) The device of claim 3, wherein the actual hazardous object is a road user or an obstacle. ### Claim 6 (depends on 3) The device of claim 3, wherein the potential hazardous object is a road user or an obstacle. ### Claim 7 (depends on 1) The device of claim 1, wherein the augmented reality display device has a head-up display or a display in at least one mirror of the transportation vehicle. ### Claim 8 (depends on 1) The device of claim 1, wherein the detection of the driving situation is based on data from a sensor device of the transportation vehicle, based on navigation information or based on information received by a data transmission unit. ### Claim 9 (depends on 1) A transportation vehicle with an augmented reality display device, wherein the transportation vehicle includes the device of claim 1 for controlling a display of the augmented reality display device. ### Claim 10 A method for controlling a display of an augmented reality display device for a transportation vehicle, the method comprising: detecting a driving situation in which a warning is to be displayed to a driver of the transportation vehicle; generating virtual objects for display by the augmented reality display device, wherein the virtual objects have moving graphical elements which simulate movement of the virtual objects; and outputting the virtual objects for display by the augmented reality display device to enable visualization of possible imminent events, actions or hazardous situations via the augmented reality display device, wherein the virtual objects displayed on the augmented reality display device include representations of actual objects in an environment surrounding the vehicle, virtual objects that indicate an increased risk of objects depicted by the virtual objects actually being present in the environment surrounding the transportation vehicle, and virtual objects providing instruction for the driver to drive the transportation vehicle to position the transportation vehicle relative to locations within the environment surrounding the transportation vehicle so as to assist the driver to follow transportation vehicle navigation instructions issued by a navigation system of the transportation vehicle or parking maneuvering assistance instructions generated by the transportation vehicle. ### Claim 11 (depends on 10) The method of claim 10, wherein the moving graphical elements form a group of light points that form a virtual object. ### Claim 12 (depends on 10) The method of claim 10, wherein at least one of the virtual objects represents an actual or potential hazardous object in the environment of the transportation vehicle. ### Claim 13 (depends on 12) The method of claim 12, wherein a position or a movement of the at least one virtual object representing the actual or potential hazardous object does not coincide with a position or movement of an actual hazardous object. ### Claim 14 (depends on 12) The method of claim 12, wherein the actual hazardous object is a road user or an obstacle. ### Claim 15 (depends on 12) The method of claim 12, wherein the potential hazardous object is a road user or an obstacle. ### Claim 16 (depends on 10) The method of claim 10, wherein the augmented reality display device has a head-up display or a display in at least one mirror of the transportation vehicle. ### Claim 17 (depends on 10) The method of claim 10, wherein the detection of the driving situation is based on data from a sensor device of the transportation vehicle, based on navigation information or based on information received by a data transmission unit. ### Claim 18 A non-transitory computer-readable storage medium with instructions which, when executed by a computer, cause the computer to execute a method for controlling a display of an augmented reality display device for a transportation vehicle, the method comprising: detecting a driving situation in which a warning is to be displayed to a driver of the transportation vehicle; generating virtual objects for display by the augmented reality display device, wherein the virtual objects have moving graphical elements which simulate movement of the virtual objects; and outputting the virtual objects for display by the augmented reality display device to enable visualization of possible imminent events, actions or hazardous situations via the augmented reality display device, wherein the virtual objects displayed on the augmented reality display device include representations of actual objects in an environment surrounding the vehicle, virtual objects that indicate an increased risk of objects depicted by the virtual objects actually being present in the environment surrounding the transportation vehicle, and virtual objects providing instruction for the driver to drive the transportation vehicle to position the transportation vehicle relative to locations within the environment surrounding the transportation vehicle so as to assist the driver to follow transportation vehicle navigation instructions issued by a navigation system of the transportation vehicle or parking maneuvering assistance instructions generated by the transportation vehicle. ### Claim 19 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the moving graphical elements form a group of light points that for a virtual object. ### Claim 20 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein at least one of the virtual objects represents an actual or potential hazardous object in the environment of the transportation vehicle. ### Claim 21 (depends on 20) The non-transitory computer readable storage medium of claim 20, wherein a position or a movement of the at least one virtual object representing the actual or potential hazardous object does not coincide with a position or movement of an actual hazardous object. ### Claim 22 (depends on 20) The non-transitory computer readable storage medium of claim 20, wherein the actual hazardous object is a road user or an obstacle. ### Claim 23 (depends on 20) The non-transitory computer readable storage medium of claim 20, wherein the potential hazardous object is a road user or an obstacle. ### Claim 24 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the augmented reality display device has a head-up display or a display in at least one mirror of the transportation vehicle. ### Claim 25 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the detection of a driving situation in which a warning for a driver of the transportation vehicle is to be displayed takes place based on data from a sensor device of the transportation vehicle, based on navigation information or based on information received by a data transmission unit.