Methods and systems for generating a semantic computation graph for understanding and grounding referring expressions

A system comprising:

• one or more hardware processors;
• and
• one or more non-transitory computer storage media storing computer-useable instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to execute operations comprising:
• parsing input text containing a referring expression referencing an image object, an object modifier, and an action performed by the image object into a hierarchical parse structure that represents the image object, the object modifier, and the action performed by the image object;
• and
• generating, based on the hierarchical parse structure, a semantic computation graph comprising an object node that represents the image object, a modifier node that represents the object modifier, and a predicate node that represents the action performed by the image object.

One or more non-transitory computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, cause the one or more processors to perform operations comprising:

• receiving input text containing at least one referring expression referencing an image object, an object modifier, and an action performed by the image object;
• parsing the at least one referring expression of the input text into a hierarchical parse structure;
• based on the hierarchical parse structure, generating a semantic computation graph comprising an object node that represents the image object, a modifier node that represents the object modifier, and a predicate node that represents the action performed by the image object;
• and
• transmitting the semantic computation graph to perform an image task.