# US 11,113,864 B2 · Generative image synthesis for training deep learning machines > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,864 B2 - Document ID: US11113864B2 - Status: Granted - Kind: B2 - Application number: 16/866,320 - Filed: 2020-05-04 - Published: 2020-08-20 - Granted: 2021-09-07 - Earliest claimed: 2017-06-27 - Assignees: The Boeing Company - Applicants: The Boeing Company - Inventors: Huafeng Yu; Tyler C. Staudinger - Web page: https://patentgrep.com/patent/US11113864B2 ## CPC - G06T 15/005: {General purpose rendering architectures} - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2215/16: Using real world measurements to influence rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A set of 3D user-designed images is used to create a high volume of realistic scenes or images which can be used for training and testing deep learning machines. The system creates a high volume of scenes having a wide variety of environmental, weather-related factors as well as scenes that take into account camera noise, distortion, angle of view, and the like. A generative modeling process is used to vary objects contained in an image so that more images, each one distinct, can be used to train the deep learning model without the inefficiencies of creating videos of actual, real life scenes. Object label data can be generated for each distinct image. This and other methods can be used to artificially create new scenes that do not have to be recorded in real-life conditions and that do not require costly and time-consuming, manual labelling or tagging of objects. ## Claims ### Claim 1 A method performed by one or more processors of a computer system, the method comprising: generating deep learning training data, wherein generating the deep learning training data includes: generating a two-dimensional (2D) scene from an electronic representation of a first three-dimensional (3D) scene of a 3D occupancy grid map; generating first label data for one or more objects of the 2D scene; executing generative modelling on the 2D scene to create one or more realistic 2D scenes; creating multiple specialized 2D scenes from each of the one or more realistic 2D scenes, each specialized 2D scene having one or more pixels that differ from each other specialized 2D scene created from each realistic 2D scene; and generating final label data for the one or more objects in the multiple specialized 2D scenes. ### Claim 2 (depends on 1) The method in claim 1, further comprising identifying an object class for each object of the one or more objects displayed in the 2D scene. ### Claim 3 (depends on 1) The method in claim 1, wherein creating the multiple specialized 2D scenes includes applying one or more climate-related conditions on the one or more realistic 2D scenes. ### Claim 4 (depends on 1) The method in claim 1, wherein creating the multiple specialized 2D scenes includes applying one or more camera distortions on the one or more realistic 2D scenes. ### Claim 5 (depends on 1) The method in claim 1, wherein at least one pixel in each realistic 2D scene of the one or more realistic 2D scenes is designated, via the first label data, an object label corresponding to an object of the one or more objects automatically without manual intervention. ### Claim 6 (depends on 1) The method in claim 1, wherein creating the one or more realistic 2D scenes includes texturizing one or more volumetric object placeholders for the one or more objects. ### Claim 7 (depends on 1) The method in claim 1, wherein multiple realistic 2D scenes are created by the generative modelling executed on the 2D scene; and wherein each realistic 2D scene of the multiple realistic 2D scenes is used to create the multiple specialized 2D scenes for that realistic 2D scene. ### Claim 8 (depends on 1) The method of claim 1, wherein the 3D occupancy grid map includes a plurality of objects including at least the one or more objects of the 2D scene. ### Claim 9 A system for generating deep learning training data, the system comprising: a computer system having program code stored thereon executable by the computer system to generate the deep learning training data by: generating a two-dimensional (2D) scene from an electronic representation of a first three-dimensional (3D) scene of a 3D occupancy grid map; generating first label data for one or more objects of the 2D scene; executing generative modelling on the 2D scene to create one or more realistic 2D scenes; creating multiple specialized 2D scenes from each of the one or more realistic 2D scenes, each specialized 2D scene having one or more pixels that differ from each other specialized 2D scene created from each realistic 2D scene; and generating final label data for the one or more objects in the multiple specialized 2D scenes. ### Claim 10 (depends on 9) The system in claim 9, wherein the program code is further executable by the computer system to identify an object class for each object of the one or more objects. ### Claim 11 (depends on 9) The system in claim 9, wherein creating the multiple specialized scenes includes applying one or more climate-related conditions on the one or more realistic 2D scenes. ### Claim 12 (depends on 9) The system in claim 9, wherein creating the multiple specialized 2D scenes includes applying one or more camera distortions on the one or more realistic 2D scenes. ### Claim 13 (depends on 9) The system in claim 9, wherein at least one pixel in each realistic 2D scene of the one or more realistic 2D scenes is designated, via the first label data, an object label corresponding to an object of the one or more objects automatically without manual intervention. ### Claim 14 (depends on 9) The system in claim 9, wherein creating the one or more multiple realistic 2D scenes includes texturizing one or more volumetric object placeholders for the one or more objects. ### Claim 15 (depends on 9) The system of claim 9, wherein the 3D occupancy grid map includes a plurality of objects including at least the one or more objects of the 2D scene. ### Claim 16 A non-transitory computer-readable storage device having program code stored thereon, the program code to be executed by one or more processors of a computer system to implement a method, the program code comprising: program code for generating deep learning training data, wherein generating deep learning training data includes: generating a two-dimensional (2D) scene from an electronic representation of a first three-dimensional (3D) scene of a 3D occupancy grid map; generating first label data for one or more objects of the 2D scene; executing generative modelling on the 2D scene to create one or more realistic 2D scenes; creating multiple specialized 2D scenes from each of the one or more realistic 2D scenes; and generating final label data for the one or more objects in the multiple specialized 2D scenes. ### Claim 17 (depends on 16) The non-transitory computer readable storage device in claim 16, wherein creating the multiple specialized 2D scenes includes applying one or more climate-related conditions on each of the one or more realistic 2D scenes. ### Claim 18 (depends on 16) The non-transitory computer readable storage device in claim 16, wherein creating the multiple specialized 2D scenes includes applying one or more camera distortions on each of the one or more realistic 2D scenes. ### Claim 19 (depends on 16) The non-transitory computer readable storage device in claim 16, wherein at least one pixel in each of the realistic 2D scenes is designated, via the first label data, an object label corresponding to an object of the one or more objects via the first label data automatically without manual intervention. ### Claim 20 (depends on 16) The non-transitory computer readable storage device of claim 16, wherein the 3D occupancy grid map includes a plurality of objects including at least the one or more objects of the 2D scene.