# US 2020/0401835 A1 · Generating scene graphs from digital images using external knowledge and image reconstruction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401835 A1
- Document ID: US20200401835A1
- Status: Published application
- Kind: A1
- Application number: 16/448,473
- Filed: 2019-06-21
- Published: 2020-12-24
- Earliest claimed: 2019-06-21
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Handong Zhao; Zhe Lin; Sheng Li; Mingyang Ling; Jiuxiang Gu
- Web page: https://patentgrep.com/patent/US20200401835A1

## CPC

- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/24323: {Tree-organised classifiers}
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/426: Graphical representations
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

## Claims

### Claim 1

A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:
determine, using object recognition processes, a plurality of object proposals for objects in a digital image;
determine, for the plurality of object proposals, a set of subgraph proposals corresponding to candidate relationships involving pairs of object proposals of the plurality of object proposals;
refine features of the plurality of object proposals and features of the set of subgraph proposals using extracted relationships corresponding to the plurality of object proposals and the set of subgraph proposals from an external knowledgebase; and
generate a semantic scene graph by predicting object labels and predicate labels based on the refined features of the plurality of object proposals and the refined features of the set of subgraph proposals.

### Claim 2 (depends on 1)

The non-transitory computer readable storage medium as recited in claim 1, further comprising instructions executed by the at least one processor, cause the computing device to:
determine feature vectors representing the features of the plurality of object proposals and feature maps representing the features of the set of subgraph proposals; and
perform an initial refinement of the feature vectors of the plurality of object proposals relative to the set of subgraph proposals and an initial refinement of the feature maps of the set of subgraph proposals relative to the plurality of object proposals using a multi-class neural network layer.

### Claim 3 (depends on 2)

The non-transitory computer readable storage medium as recited in claim 2, wherein the instructions that cause the computing device to refine the features of the plurality of object proposals and the features of the set of subgraph proposals cause the computing device to:
determine a predetermined number of relationships that occur most frequently in the external knowledgebase for an identified object proposal of the plurality of object proposals; and
encode, using a recurrent neural network, word embeddings comprising the predetermined number of relationships with the identified object proposal.

### Claim 4 (depends on 3)

The non-transitory computer readable storage medium as recited in claim 3, wherein the instructions that cause the computing device to refine the features of the plurality of object proposals and the features of the set of subgraph proposals cause the computing device to jointly refine the feature vectors and the feature maps utilizing episodic memory states in a dynamic memory network.

### Claim 5 (depends on 1)

The non-transitory computer readable storage medium as recited in claim 1, wherein each subgraph proposal of the set of subgraph proposals corresponds to a pair of object proposals of the plurality of object proposals, and wherein each object proposal of the plurality of object proposals is associated with a subset of subgraph proposals of the set of subgraph proposals.

### Claim 6 (depends on 1)

The non-transitory computer readable storage medium as recited in claim 1, wherein the instructions that cause the computing device to determine the set of subgraph proposals cause the computing device to:
determine a first score for a first object proposal and a second score for a second object proposal in an identified pair of object proposals; and
determine a subgraph proposal for the identified pair of object proposals by determining a union box with a confidence score as a product of the first score and the second score.

### Claim 7 (depends on 1)

The non-transitory computer readable storage medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a synthesized image based on the object labels used to generate the semantic scene graph;
determine a difference between the digital image and the synthesized image; and
modify, using the determined difference, one or more parameters of a scene graph generation model that generates the semantic scene graph.

### Claim 8 (depends on 7)

The non-transitory computer readable storage medium as recited in claim 7, wherein the instructions that cause the computing device to generate the synthesized image cause the computing device to generate, using a cascaded refinement network, the synthesized image comprising objects from a scene layout based on the object labels and locations associated with the object labels.

### Claim 9 (depends on 7)

The non-transitory computer readable storage medium as recited in claim 7, wherein the instructions that cause the computing device to determine the difference between the digital image and the synthesized image cause the computing device to determine, using a generative adversarial network, a loss associated with the synthesized image relative to the digital image.

### Claim 10 (depends on 9)

The non-transitory computer readable storage medium as recited in claim 9, wherein the instructions that cause the computing device to modify the one or more parameters of the scene graph generation model that generates the semantic scene graph cause the computing device to modify, based on the determined loss and using backpropagation, one or more parameters of an object detection model used to determine the plurality of object proposals for objects in the digital image.

### Claim 11

In a digital medium environment, a system for generating accurate scene graph representations of digital images, the system comprising:
at least one processor; and
a non-transitory computer memory comprising instructions that, when executed by the at least one processor, cause the system to:
determine, using a machine-learning model comprising object recognition processes, a plurality of object proposals for objects in a digital image by estimating objects and bounding boxes for the estimated objects within the digital image;
determine, for pairs of object proposals of the plurality of object proposals, a set of subgraph proposals representing candidate relationships involving the pairs of object proposals, wherein determining a subgraph proposal of the set of subgraph proposals comprises determining a product of confidence scores of object proposals within a pair of object proposals;
extract, from an external knowledgebase, relationships corresponding to semantic entities matching the plurality of object proposals;
refine features of the plurality of object proposals based on encoded versions of the extracted relationships using a dynamic memory network; and
generate a semantic scene graph by predicting object labels and predicate labels based on the refined features of the plurality of object proposals and refined features of the set of subgraph proposals.

### Claim 12 (depends on 11)

The system as recited in claim 11, wherein the instructions that cause the system to extract, from the external knowledgebase, relationships corresponding to semantic entities matching the plurality of object proposals cause the system to determine a predetermined number of relationships that occur most frequently in the external knowledgebase for an identified object proposal of the plurality of object proposals.

### Claim 13 (depends on 12)

The system as recited in claim 12, wherein the instructions that cause the system to refine the features of the plurality of object proposals based on encoded versions of the extracted relationships using the dynamic memory network cause the system to generate, using a recurrent neural network, the encoded versions based on word embeddings comprising the predetermined number of relationships with the identified object proposal.

### Claim 14 (depends on 13)

The system as recited in claim 13, wherein the instructions that cause the system to refine the features of the plurality of object proposals based on encoded versions of the extracted relationships using the dynamic memory network cause the system to jointly refine the features of the plurality of object proposals and features of the set of subgraph proposals utilizing episodic memory states in a dynamic memory network in connection with the encoded versions of the extracted relationships from the external knowledgebase.

### Claim 15 (depends on 11)

The system as recited in claim 11, wherein the instructions that cause the system to predict the object labels and the predicate labels cause the system to:
predict the object labels directly from feature vectors representing the plurality of object proposals; and
predict the predicate labels based on feature vectors for pairs of object proposals of the plurality of object proposals and corresponding feature maps representing subgraph proposals of the set of subgraph proposals.

### Claim 16 (depends on 11)

The system as recited in claim 11, further comprising instructions that, when executed by the at least one processor, cause the system to:
generate a synthesized image based on the semantic scene graph;
determine a difference between the digital image and the synthesized image; and
modify, using the determined difference, one or more parameters of a scene graph generation model that generates the semantic scene graph.

### Claim 17 (depends on 16)

The system as recited in claim 16, wherein the instructions that cause the system to generate the synthesized image cause the system to generate, using a cascaded refinement network, the synthesized image comprising objects from a scene layout based on the object labels and locations associated with the plurality of object proposals in the semantic scene graph.

### Claim 18 (depends on 16)

The system as recited in claim 16, wherein the instructions that cause the system to:
determine the difference between the digital image and the synthesized image cause the system to determine, using a generative adversarial network, a loss associated with the synthesized image relative to the digital image; and
modify the one or more parameters of the scene graph generation model that generates the semantic scene graph cause the system to modify, based on the determined loss and using backpropagation, one or more parameters an object detection model used to determine the plurality of object proposals for objects in the digital image.

### Claim 19

In a digital medium environment, a method of generating accurate scene graph representations of digital images, the method comprising:
determine, using object recognition processes, a plurality of object proposals for objects in a digital image and a set of subgraph proposals for each pair of object proposals of the plurality of object proposals;
a step for refining features of the plurality of object proposals and features of the set of subgraph proposals using an external knowledgebase; and
generating, using a scene graph generation model, a semantic scene graph by predicting object labels and predicate labels based on the refined features of the plurality of object proposals and the refined features of the set of subgraph proposals.

### Claim 20 (depends on 19)

The method as recited in claim 19, further comprising:
generating a synthesized image based on the semantic scene graph;
determining a loss between the digital image and the synthesized image; and
modifying, using the determined loss, one or more parameters of a scene graph generation model that generates the semantic scene graph.