Spatial-temporal graph-to-sequence learning based grounded video descriptions

A system comprising one or more computer processors and a memory containing a program which when executed by the computer processors performs an operation comprising:

• generating an initial graph representing a plurality of object features in a plurality of frames of a received video input;
• generating an implicit graph for the plurality of object features in the plurality of frames using a similarity function;
• combining the initial graph and the implicit graph to form a refined graph;
• processing the refined graph, using attention processes, to generate an attended hierarchical graph of the plurality of object features for the plurality of frames;
• and
• generating a grounded video description for the received video input using at least the hierarchical graph of the plurality of features.

A computer program product for water bottle rental and sharing, the computer program product comprising:

• a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:
• generating an initial graph representing a plurality of object features in a plurality of frames of a received video input;
• generating an implicit graph for the plurality of object features in the plurality of frames using a similarity function;
• combining the initial graph and the implicit graph to form a refined graph;
• processing the refined graph, using attention processes, to generate an attended hierarchical graph of the plurality of object features for the plurality of frames;
• and
• generating a grounded video description for the received video input using at least the hierarchical graph of the plurality of features.