# US 2021/0275918 A1 · Unsupervised learning of scene structure for synthetic data generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0275918 A1 - Document ID: US20210275918A1 - Status: Published application - Kind: A1 - Application number: 17/117,425 - Filed: 2020-12-10 - Published: 2021-09-09 - Earliest claimed: 2020-03-06 - Assignees: NVIDIA Corporation - Applicants: Nvidia Corporation - Inventors: Jeevan Devaranjan; Sanja Fidler; Amlan Kar - Web page: https://patentgrep.com/patent/US20210275918A1 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - A63F 13/52: involving aspects of the displayed game scene - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/54: Browsing; Visualisation therefor - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/092: Reinforcement learning - G06N 3/094: Adversarial learning - G06N 5/025: Extracting rules from data - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 15/00: Three-dimensional [3D] image rendering - G06T 15/205: {Image-based rendering} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/20: in augmented reality scenes - A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor - A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/048: Activation functions - G06N 20/00: Machine learning - G06T 2210/61: Scene description - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A rule set or scene grammar can be used to generate a scene graph that represents the structure and visual parameters of objects in a scene. A renderer can take this scene graph as input and, with a library of content for assets identified in the scene graph, can generate a synthetic image of a scene that has the desired scene structure without the need for manual placement of any of the objects in the scene. Images or environments synthesized in this way can be used to, for example, generate training data for real world navigational applications, as well as to generate virtual worlds for games or virtual reality experiences. ## Claims ### Claim 1 A computer-implemented method, comprising: generating a scene structure using at least a subset of a plurality of rules from a rule set, the plurality of rules specifying relationships between types of objects in a scene; determining one or more parameter values for one or more of the objects represented in the scene structure; generating a scene graph based on the scene structure and including the parameter values; and providing the scene graph to a rendering engine to render an image of the scene. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: providing, to the rendering engine, a library of object-related content to be used to render the objects represented in the scene structure using the one or more parameter values for the objects. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: causing the rendering engine to include at least one of object labels or scene structure information with the image of the scene, wherein the image is capable of being used as training data for training a neural network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving input indicating that one of the rules or objects is not to be utilized for the scene; updating the scene graph to reflect the input; and providing the updated scene graph to the rendering engine to render an updated image of the scene. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: utilizing the rule set to generate a diverse set of scene graphs; and generating a virtual environment using the diverse set of scene graphs. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the virtual environment is a gaming environment, and wherein the rendering engine is to use the diverse set of scene graphs to render one or more images of the gaming environment for one or more players of a game corresponding to the gaming environment. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: utilizing the virtual environment in one or more testing simulations for one or more autonomous or semi-autonomous machines. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the scene structure is generated from the rule set in an unsupervised manner without data annotation on one or more input images. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: assigning probabilities to rules of the rule set; and selecting the subset of the plurality of rules by sampling the rules according to the probabilities. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, further comprising: determining a scene mask to indicate which one or more rules of the rule set are able to be selected through the sampling. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, further comprising: generating the scene structure using an iterative process in which, at one or more of a plurality of time steps, a determination is made whether to perform expansion for one or more object types in the scene structure, the object types being positioned at varying levels in a hierarchical scene structure. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein a generative model is used to generate the scene structure using the rule set. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, further comprising: generating a latent vector as input to the generative model, the latent vector having a length equal to a number of rules in the rule set. ### Claim 14 A system, comprising: at least one processor; and memory including instructions that, when executed by the at least one processor, cause the system to: generate a scene structure using at least a subset of a plurality of rules from a rule set, the plurality of rules specifying relationships between types of objects in a scene; determine one or more parameter values for one or more of the objects represented in the scene structure; generate a scene graph based on the scene structure and including the parameter values; and provide the scene graph to a rendering engine to render an image of the scene. ### Claim 15 (depends on 14) The system of claim 14, wherein the instructions when executed further cause the system to: cause the rendering engine to include at least one of object labels or scene structure information with the image of the scene, wherein the image is capable of being used as training data for training a neural network. ### Claim 16 (depends on 14) The system of claim 14, wherein the instructions when executed further cause the system to: receive input indicating that at least one of the rules or objects is not to be utilized for the scene; update the scene graph to reflect the input; and provide the updated scene graph to the rendering engine to render an updated image of the scene. ### Claim 17 (depends on 14) The system of claim 14, wherein the instructions when executed further cause the system to: utilize the rule set to generate a diverse set of scene graphs; and generate a virtual environment using the diverse set of scene graphs. ### Claim 18 (depends on 14) The system of claim 14, wherein the system comprises at least one of: a system for performing graphical rendering operations; a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing deep learning operations; a system implemented using an edge device; a system incorporating one or more Virtual Machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. ### Claim 19 A method for synthesizing training data, comprising: obtaining a rule set including a plurality of rules specifying relationships between types of objects in a scene; generating a plurality of scene structures based on the plurality of rules; determining one or more parameter values for each of the objects represented in the plurality of scene structures; generating a plurality of scene graphs based on the plurality of scene structures and including the parameter values; and providing the plurality of scene graphs to a rendering engine to render a set of images including data for the objects represented in the images, the set of images representing training data for use in training one or more neural networks. ### Claim 20 (depends on 19) The method of claim 19, further comprising: providing, to the rendering engine, a library of object-related content to be used to render the objects represented in the scene structure using the one or more parameter values for the objects. ### Claim 21 (depends on 19) The method of claim 19, further comprising: receiving input indicating that one of the rules or objects is not to be utilized; updating the plurality of scene graphs to reflect the input; and providing the updated scene graphs to the rendering engine to render an updated set of images. ### Claim 22 (depends on 19) The method of claim 19, wherein the scene structure is generated from the rule set in an unsupervised manner without data annotation on one or more input images. ### Claim 23 (depends on 19) The method of claim 19, further comprising: assigning probabilities to rules of the rule set; and selecting a subset of the plurality of rules for respective scene structures by sampling the rules according to the probabilities.