# US 12,112,432 B2 · Realistic 3D virtual world creation and simulation for training automated driving systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,432 B2 - Document ID: US12112432B2 - Status: Granted - Kind: B2 - Application number: 17/885,633 - Filed: 2022-08-11 - Published: 2022-12-01 - Granted: 2024-10-08 - Earliest claimed: 2016-06-28 - Assignees: Cognata Ltd. - Applicants: Cognata Ltd. - Inventors: Dan Atsmon - Web page: https://patentgrep.com/patent/US12112432B2 ## CPC - G06T 17/05: Geographic models - G01C 11/04: Interpretation of pictures - G01C 21/3815: {Road data} - G01C 21/3826: {Terrain data} - G01C 21/3867: {Geometry of map features, e.g. shape points, polygons or for simplified maps} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G05D 1/81: Handing over between on-board automatic and on-board manual control - G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer implemented method of creating a simulated realistic virtual model of a geographical area for training an autonomous driving system, comprising obtaining geographic map data of a geographical area, obtaining visual imagery data of the geographical area, classifying static objects identified in the visual imagery data to corresponding labels to designate labeled objects, superimposing the labeled objects over the geographic map data, generating a virtual 3D realistic model emulating the geographical area by synthesizing a corresponding visual texture for each of the labeled objects and injecting synthetic 3D imaging feed of the realistic model to imaging sensor(s) input(s) of the autonomous driving system controlling movement of an emulated vehicle in the realistic model where the synthetic 3D imaging feed is generated to depict the realistic model from a point of view of emulated imaging sensor(s) mounted on the emulated vehicle. ## Claims ### Claim 1 A system for creating a simulated virtual realistic model of a geographical area for training, evaluating or validating an autonomous driving system (ADS), comprising: at least one processor adapted to execute code, the code comprising: code instructions to obtain geographic map data of a geographical area; code instructions to obtain visual imagery data of the geographical area; code instructions to classify a plurality of static objects identified in the visual imagery data to corresponding labels to designate a plurality of labeled objects; code instructions to superimpose the plurality of labeled objects over the geographic map data; code instructions to generate a virtual three dimensional (3D) realistic ADS training model emulating the geographical area by synthesizing a corresponding visual texture for each of the plurality of labeled objects; and code instructions to create the simulated virtual realistic model by inserting one or more dynamic objects to the virtual 3D realistic ADS training model; and code instructions to apply movement of said one or more dynamic objects according to movement patterns predefined or learned for said geographical area. ### Claim 2 (depends on 1) A system according to claim 1, wherein said one or more dynamic objects include one or more of a ground vehicle, an aerial vehicle, a naval vehicle, a pedestrian, an animal, vegetation and a dynamically changing road infrastructure object. ### Claim 3 (depends on 1) A system according to claim 1, wherein said dynamic objects are simulated dynamic objects that mimic real world dynamic objects behavior. ### Claim 4 (depends on 1) A system according to claim 1, wherein said movement emulates real world traffic in said geographical area for training, evaluating or validating the autonomous driving system. ### Claim 5 (depends on 1) A system according to claim 1, wherein at least one of said one or more dynamic objects comprises a vehicle and the movement of said vehicle is according to a driver behavior pattern identified through big-data analytics of vehicles movement in said geographical area. ### Claim 6 (depends on 1) A system according to claim 1, wherein at least one of said one or more dynamic objects comprises a vehicle and the movement of said vehicle is according to one more driver behavior classes or patterns identified at the geographical area, the driver behavior classes including one or more of an aggressive driver, a normal driver, a patient driver or a reckless driver. ### Claim 7 (depends on 1) A system according to claim 1, wherein said one or more dynamic objects include a plurality of vehicles and the movement is according to driver behavior classes according to a density function calculated for the geographical area which represents the distribution of driver classes detected in the geographical area replicated by the virtual realistic model. ### Claim 8 (depends on 7) A system according to claim 7, wherein different driving behavior patterns are calculated for different weather conditions. ### Claim 9 (depends on 1) A system according to claim 1, wherein at least one of said one or more dynamic objects is a traffic light and the movement of said traffic light comprises controlling switching of lights in accordance with traffic control directives applicable to traffic lights in said geographical area. ### Claim 10 (depends on 1) A system according to claim 1, wherein the processor further comprises: code instructions to adjust the virtual three dimensional (3D) realistic ADS training model according to one or more environmental characteristics, including one or more of a lighting condition, a weather condition attribute or timing attribute. ### Claim 11 (depends on 10) A system according to claim 10, wherein the environmental characteristics are typical to said geographical area. ### Claim 12 A computer implemented method of creating a simulated realistic virtual model of a geographical area for training an autonomous driving system, comprising: obtaining geographic map data of a geographical area; obtaining visual imagery data of the geographical area; classifying a plurality of static objects identified in the visual imagery data to corresponding labels to designate a plurality of labeled objects; superimposing the plurality of labeled objects over the geographic map data; generating a virtual three dimensional (3D) realistic ADS training model emulating the geographical area by synthesizing a corresponding visual texture for each of the plurality of labeled objects; and creating the simulated virtual realistic model by inserting one or more dynamic objects to the virtual 3D realistic ADS training model; and applying movement of said one or more dynamic objects according to movement patterns predefined or learned for said geographical area. ### Claim 13 (depends on 12) A computer implemented method according to claim 12, wherein said one or more dynamic objects include one or more of a ground vehicle, an aerial vehicle, a naval vehicle, a pedestrian, an animal, vegetation and a dynamically changing road infrastructure object. ### Claim 14 (depends on 12) A computer implemented method according to claim 12, wherein said dynamic objects are simulated dynamic objects that mimic real world dynamic objects behavior. ### Claim 15 (depends on 12) A computer implemented method according to claim 12, wherein at least one of said one or more dynamic objects comprises a vehicle and the movement of said vehicle is according to a driver behavior pattern identified through big-data analytics of vehicles movement in said geographical area. ### Claim 16 (depends on 12) A computer implemented method according to claim 12, wherein at least one of said one or more dynamic objects comprises a vehicle and the movement of said vehicle is according to one more driver behavior classes or patterns identified at the geographical area, the driver behavior classes including one or more of an aggressive driver, a normal driver, a patient driver or a reckless driver. ### Claim 17 (depends on 12) A computer implemented method according to claim 12, wherein said one or more dynamic objects include a plurality of vehicles and the movement is according to driver behavior classes according to a density function calculated for the geographical area which represents the distribution of driver classes detected in the geographical area replicated by the virtual realistic model. ### Claim 18 (depends on 17) A computer implemented method according to claim 17, wherein different driving behavior patterns are calculated for different weather conditions. ### Claim 19 (depends on 12) A computer implemented method according to claim 12, wherein at least one of said one or more dynamic objects is a traffic light and the movement of said traffic light comprises controlling switching of lights in accordance with traffic control directives applicable to traffic lights in said geographical area. ### Claim 20 (depends on 12) A computer implemented method according to claim 12, further comprising: adjusting the virtual three dimensional (3D) realistic ADS training model according to one or more environmental characteristics typical to said geographical area, including one or more of a lighting condition, a weather condition attribute or timing attribute.