Conditioned 3d layout prediction

A computer-implemented method for machine-learning a machine-learning function for applying machine-learning, comprising:

• obtaining a dataset of ground truth 3D layouts, each ground truth 3D layout representing a respective scene and including:
• a respective floorplan,
• a respective 3D arrangement of one or more 3D bounding boxes, each bounding box being defined by a value of a predetermined set of one or more bounding boxes parameters, and
• for each 3D bounding box, a respective semantic category;
• obtaining a probability distribution of noise levels;
• for each ground truth 3D layout, obtaining a respective perturbed 3D layout computable by perturbing at least one bounding box parameter of at least one 3D bounding box of the ground truth 3D layout, the perturbing including:
• sampling a respective noise level based on the probability distribution, and
• for each respective bounding box parameter of the at least one bounding box parameter:
• sampling a respective noise value based on the respective noise level, and
• applying the respective noise value to the respective bounding box parameter;
• and
• training the function over the dataset based on a loss which penalizes a dissimilarity metric between each ground truth 3D layout and a respective predicted 3D layout obtainable by applying the function to the respective perturbed 3D layout,
• wherein the training is performed with a conditioning drop-out with respect to at least one layout parameter, the at least one layout parameter including the floorplan and/or the semantic category for each 3D bounding box, and
• wherein the applying machine-learning includes:
• obtaining the machine-learning function preconfigured to take an input 3D layout and a given noise level, a 3D layout having a set of layout parameters including a floorplan, a 3D arrangement of one or more 3D bounding boxes, each bounding box being defined in the 3D arrangement by a value of a predetermined set of one or more bounding box parameters, and a semantic category for each 3D bounding box, the input 3D layout including:
• a given floorplan,
• a first 3D arrangement of one or more given 3D bounding boxes, each bounding box being defined in the first 3D arrangement by a first value of the predetermined set of one or more bounding box parameters, and
• for each given 3D bounding box, a given semantic category,
• wherein the machine-learning function is further preconfigured to predict an output 3D layout, the output 3D layout including:
• the given floorplan,
• a second 3D arrangement of the one or more given 3D bounding boxes, each bounding box being defined in the second 3D arrangement by a second value of the predetermined set of one or more bounding box parameters, and
• for each given 3D bounding box, the given semantic category, and
• wherein the function is configured to predict a second value of the predetermined set of one or more bounding box parameters different from the first value of the predetermined set of one or more bounding box parameters, and
• wherein the function is further preconfigured with a conditioning drop-out with respect to at least one layout parameter, the at least one layout parameter including the floorplan and/or the semantic category for each 3D bounding box;
• obtaining a set of conditioning inputs, each conditioning input including a distinct value of one layout parameter among the at least one layout parameter, and a same value of each other layout parameter among the floorplan and the semantic category for each 3D bounding box;
• for each conditioning input, determining one or more conditioning candidate 3D layouts, each conditioning candidate 3D layout being a result of an iterative sampling with the preconfigured function;
• for each condition input, determining a plurality of perturbed conditioning candidate 3D layouts, each by adding a respective noise to a respective conditioning candidate 3D layout;
• for each perturbed conditioning candidate 3D layout, applying the preconfigured function to the perturbed conditioning candidate 3D layout, in which the one layout parameter is dropped out, thereby obtaining a respective unconditional output;
• for each perturbed conditioning candidate 3D layout, computing a reconstruction error between the respective conditioning candidate 3D layout and the respective unconditional output;
• and
• averaging the reconstruction errors over the plurality of perturbed conditioning candidate 3D layouts, thereby obtaining a score.

A device comprising:

• a processor;
• and
• a non-transitory computer-readable data storage medium having recorded thereon:
• a first computer program having instructions for applying a machine learning function that when executed by the processor causes the processor to be configured to:
• obtain a machine-learning function preconfigured to take an input 3D layout and a given noise level, a 3D layout having a set of layout parameters including a floorplan, a 3D arrangement of one or more 3D bounding boxes, each bounding box being defined in the 3D arrangement by a value of a predetermined set of one or more bounding box parameters, and a semantic category for each 3D bounding box, the input 3D layout including:
• a given floorplan,
• a first 3D arrangement of one or more given 3D bounding boxes, each bounding box being defined in the first 3D arrangement by a first value of the predetermined set of one or more bounding box parameters, and
• for each given 3D bounding box, a given semantic category,
• wherein the machine-learning function is further preconfigured to predict an output 3D layout, the output 3D layout including:
• the given floorplan,
• a second 3D arrangement of the one or more given 3D bounding boxes, each bounding box being defined in the second 3D arrangement by a second value of the predetermined set of one or more bounding box parameters, and
• for each given 3D bounding box, the given semantic category, and
• wherein the function is configured to predict a second value of the predetermined set of one or more bounding box parameters different from the first value of the predetermined set of one or more bounding box parameters, and
• wherein the function is further preconfigured with a conditioning drop-out with respect to at least one layout parameter, the at least one layout parameter including the floorplan and/or the semantic category for each 3D bounding box;
• obtain a set of conditioning inputs, each conditioning input including a distinct value of one layout parameter among the at least one layout parameter, and a same value of each other layout parameter among the floorplan and the semantic category for each 3D bounding box;
• for each conditioning input, determine one or more conditioning candidate 3D layouts, each conditioning candidate 3D layout being a result of an iterative sampling with the preconfigured function;
• for each condition input, determine a plurality of perturbed conditioning candidate 3D layouts, each by adding a respective noise to a respective conditioning candidate 3D layout;
• for each perturbed conditioning candidate 3D layout, apply the preconfigured function to the perturbed conditioning candidate 3D layout, in which the one layout parameter is dropped out, thereby obtaining a respective unconditional output;
• for each perturbed conditioning candidate 3D layout, compute a reconstruction error between the respective conditioning candidate 3D layout and the respective unconditional output;
• and
• average the reconstruction errors over the plurality of perturbed conditioning candidate 3D layouts, thereby obtaining a score;
• a second computer program having instructions for machine-learning the machine-learning function that when executed by the processor causes the processor to be configured to:
• obtain a dataset of ground truth 3D layouts, each ground truth 3D layout representing a respective scene and including:
• a respective floorplan,
• a respective 3D arrangement of one or more 3D bounding boxes, each bounding box being defined by a value of a predetermined set of one or more bounding boxes parameters, and
• for each 3D bounding box, a respective semantic category;
• obtain a probability distribution of noise levels;
• for each ground truth 3D layout, obtain a respective perturbed 3D layout computable by perturbing at least one bounding box parameter of at least one 3D bounding box of the ground truth 3D layout, the perturbing including:
• sampling a respective noise level based on the probability distribution;
• and
• for each respective bounding box parameter of the at least one bounding box parameter, sample a respective noise value based on the respective noise level;
• for each respective bounding box parameter of the at least one bounding box parameter, applying the respective noise value to the respective bounding box parameter;
• training the function over the dataset based on a loss which penalizes a dissimilarity metric between each ground truth 3D layout and a respective predicted 3D layout obtainable by applying the function to the respective perturbed 3D layout,
• wherein the training is performed with a conditioning drop-out with respect to at least one layout parameter, the at least one layout parameter including the floorplan and/or the semantic category for each 3D bounding box.