Layered image understanding

In a computing environment, a system, comprising:

• at least one processor;
• and
• an image understanding mechanism implemented on the at least one processor and configured to:
• compare a query image with annotated image data associated with sample images to identify a set of valid moves toward at least one sample image that comprises at least one of a scene modification or a transformation, the set of valid moves based upon support constraints of a semantic structure of objects;
• execute the set of valid moves to interpolate a synthesized image from the query image and corresponding objects from the annotated image data, including determining a depth layer of at least one inserted object in the synthesized image by using the semantic segments and relative depth information to match a depth distribution of objects in layers above and below the at least one inserted object in the at least one sample image to the depth distribution of objects in layers above and below the at least one inserted object in the synthesized image;
• and
• determine whether the synthesized image maximizes an objective corresponding to the query image.

One or more computer storage devices having computer-executable instructions, which on execution by a computer, cause the computer to perform operations, comprising:

• an image understanding mechanism initializing a query image representing a scene;
• the image understanding mechanism searching a set of scenes corresponding to a set of sample images in a scene network to identify one or more objects from the set of sample images for comparing with the scene represented by the query image;
• and
• the image understanding mechanism determining whether there are one or more valid scene modification moves for synthesizing a scene representation for the query image, including by transforming the query image and the one or more objects into semantic segments comprising at least a portion of the scene representation to recombine the one or more objects into the scene representation as a synthesized image, including determining a depth layer of at least one inserted object in the synthesized image by using the semantic segments and relative depth information to match a depth distribution of objects in layers above and below the at least one inserted object in a corresponding sample image to the depth distribution of objects in layers above and below the at least one inserted object in the synthesized image.