Generating scene graphs from digital images using external knowledge and image reconstruction

In a digital medium environment, a system for generating accurate scene graph representations of digital images, the system comprising:

• at least one processor;
• and
• a non-transitory computer memory comprising instructions that, when executed by the at least one processor, cause the system to:
• determine, using a machine-learning model comprising object recognition processes, a plurality of object proposals for objects in a digital image by estimating objects and bounding boxes for the estimated objects within the digital image;
• determine, for pairs of object proposals of the plurality of object proposals, a set of subgraph proposals representing candidate relationships involving the pairs of object proposals, wherein determining a subgraph proposal of the set of subgraph proposals comprises determining a product of confidence scores of object proposals within a pair of object proposals;
• extract, from an external knowledgebase, relationships corresponding to semantic entities matching the plurality of object proposals;
• refine features of the plurality of object proposals based on encoded versions of the extracted relationships using a dynamic memory network;
• and
• generate a semantic scene graph by predicting object labels and predicate labels based on the refined features of the plurality of object proposals and refined features of the set of subgraph proposals.

In a digital medium environment, a method of generating accurate scene graph representations of digital images, the method comprising:

• determine, using object recognition processes, a plurality of object proposals for objects in a digital image and a set of subgraph proposals for each pair of object proposals of the plurality of object proposals;
• a step for refining features of the plurality of object proposals and features of the set of subgraph proposals using an external knowledgebase;
• and
• generating, using a scene graph generation model, a semantic scene graph by predicting object labels and predicate labels based on the refined features of the plurality of object proposals and the refined features of the set of subgraph proposals.