# US 9,517,415 B2 · Method and system for generating augmented reality with a display of a motor vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,517,415 B2 - Document ID: US9517415B2 - Status: Granted - Kind: B2 - Application number: 13/722,819 - Filed: 2012-12-20 - Published: 2013-06-27 - Granted: 2016-12-13 - Earliest claimed: 2011-12-21 - Assignees: Harman Becker Automotive Systems GmbH - Applicants: Harman Becker Automotive Systems GmbH - Inventors: Tobias Muench; Philipp Schmauderer; Christoph Benz; Andreas Koerner - Web page: https://patentgrep.com/patent/US9517415B2 ## CPC - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/655: by importing photos, e.g. of the player - A63F 13/52: involving aspects of the displayed game scene - A63F 13/803: Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A63F 13/216: using geographical information, e.g. location of the game device or player using GPS - A63F 13/525: Changing parameters of virtual cameras - A63F 2300/205: for detecting the geographical location of the game platform - A63F 2300/6661: for changing the position of the virtual camera - A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system for generating augmented reality with a display of a vehicle is provided. The method comprises recording an image of an environment of the vehicle in the form of image data, determining a virtual space in three dimensions from the image data, and detecting a real object in the image data. The method further comprises determining a first coordinate range of the real object in the three dimensional virtual space, adding a virtual element to the three dimensional virtual space, and controlling the virtual element in the three dimensional virtual space based on a user input. The method further comprises outputting the environment and the controlled virtual element in combined form in an output image, modifying the output image when the first coordinate range of the real object and a second coordinate range of the controlled virtual element form an intersection area, and displaying the output image. ## Claims ### Claim 1 A method for generating an augmented reality with a display of a motor vehicle, the method comprising the steps of: recording a stereoscopic image of an environment of the motor vehicle at a 360° angle with the aid of a plurality of cameras of the motor vehicle in the form of stereoscopic image data; detecting a three-dimensional course of a road from the stereoscopic image data; determining a virtual space in three dimensions from the detected three-dimensional course of the road; detecting a real object in the stereoscopic image data; determining a first coordinate range of the real object in the three dimensional virtual space; adding a virtual element to the three dimensional virtual space, the virtual element having a second coordinate range; controlling the virtual element in the three dimensional virtual space based on a user input; outputting the environment and the controlled virtual element in combined form in an output image; modifying the output image when the first coordinate range of the real object and the second coordinate range of the controlled virtual element form an intersection area; projecting the output image onto a window pane of the motor vehicle from certain angles by means of the display of the motor vehicle so that the output image is not visible to a driver of the motor vehicle; and implementing a video game in the three dimensional virtual space, wherein the virtual element is controlled according to a ballistic trajectory in the three dimensional virtual space. ### Claim 2 (depends on 1) The method according to claim 1, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space. ### Claim 3 (depends on 1) The method according to claim 1, wherein the output image is modified by supplementing or replacing object image data of the real object with a virtual object. ### Claim 4 (depends on 1) The method according to claim 1, wherein the output image is modified by generating a virtual object from object image data of the real object and outputting the virtual object in the output image. ### Claim 5 (depends on 1) The method according to claim 1, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position. ### Claim 6 An augmented reality system of a motor vehicle, the system comprising: an image capture device with a plurality of cameras which is configured to record a stereoscopic image of an environment of the motor vehicle at a 360° angle in the form of stereoscopic image data; an arithmetic unit which is connected to the image capture device; a control unit which is connected to the arithmetic unit; and a display which is connected to the arithmetic unit displaying an output image; wherein the arithmetic unit is configured to: detect a three-dimensional course of a road from the stereoscopic image data; determine a three dimensional virtual space from the detected three-dimensional course of the road; detect a real object in the stereoscopic image data; determine a first coordinate range of the real object in the three dimensional virtual space; add a virtual element having a second coordinate range to the three dimensional virtual space; control the second coordinate range of the virtual element in the three dimensional virtual space by means of the control unit based on a user input; combine the environment and the controlled virtual element in the output image; modify the output image when the first coordinate range of the real object and the second coordinate range of the controlled virtual element form an intersection area; and implement a video game in the three dimensional virtual space, wherein the virtual element is controlled according to a ballistic trajectory in the three dimensional virtual space, and wherein the display comprises a projector for projecting the output image onto a window pane from certain angles so that the output image is not visible to a driver of the motor vehicle. ### Claim 7 (depends on 6) The system according to claim 6, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space. ### Claim 8 (depends on 6) The system according to claim 6, wherein the output image is modified by one of supplementing and replacing object image data of the real object with a virtual object. ### Claim 9 (depends on 6) The system according to claim 6, wherein the output image is modified by generating a virtual object from object image data of the real object and outputting the virtual object in the output image. ### Claim 10 (depends on 6) The system according to claim 6, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position. ### Claim 11 An augmented reality system in a vehicle, the system comprising: an image capture device with a plurality of cameras configured to record a stereoscopic image of an environment about the vehicle at a 360° angle to provide stereoscopic image data; a display for providing an output image; and an arithmetic unit connected to the image capture device and to the display, the arithmetic unit being configured to: detect a three-dimensional course of a road from the stereoscopic image data; determine a three dimensional virtual space from the detected three-dimensional course of the road; detect a real object in the stereoscopic image data; determine a first coordinate range of the real object in the three dimensional virtual space; add a virtual element having a second coordinate range to the three dimensional virtual space; control the second coordinate range of the virtual element in the three dimensional virtual space based on a user input; combine the environment and the controlled virtual element in the output image; and modify the output image when the first coordinate range of the real object and the controlled second coordinate range of the virtual element form an intersection area; and implement a video game in the three dimensional virtual space, wherein the virtual element is controlled according to a ballistic trajectory in the three dimensional virtual space, and wherein the display comprises a projector for projecting the output image onto a window pane from certain angles so that the output image is not visible to a driver of the vehicle. ### Claim 12 (depends on 11) The system of claim 11, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space. ### Claim 13 (depends on 11) The system of claim 11, wherein the output image is modified by one of supplementing and replacing object image data of the real object with a virtual object. ### Claim 14 (depends on 11) The system of claim 11, wherein the output image is modified by generating a virtual object from object image data of the real object and outputting the virtual object in the output image. ### Claim 15 (depends on 11) The system of claim 11, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position.