Graph-based video instance segmentation

A system comprising:

• a processor;
• and
• a memory device configured to store a media clip including a plurality of consecutive video frames and a graph of nodes and edges based on a reference instance of an object in the media clip, the memory device further storing a media editing application executable by the processor for performing operations comprising;
• identifying the reference instance of the object in a reference frame of the media clip;
• identifying features in a target frame of the media clip;
• producing the graph of nodes and edges based on the reference instance of the object in the reference frame and the features in the target frame, wherein each node of the graph represents a feature in the target frame or the reference instance of the object in the reference frame and each edge of the graph represents a spatiotemporal relationship between the feature in the target frame and the reference instance of the object in the reference frame;
• iteratively updating embeddings of the nodes and edges of the graph based on the spatiotemporal relationship to produce fused node embeddings and updated edge embeddings;
• and
• detecting a target instance of the object in the target frame based on the fused node embeddings.

A non-transitory computer-readable medium storing program code executable by a processor to perform operations, the operations comprising:

• identifying a reference instance of an object in a reference frame of a media clip including a plurality of consecutive video frames;
• identifying features in a target frame of the media clip in the feature domain;
• a step for producing a graph of nodes and edges based on the reference instance of the object in the reference frame and the features in the target frame;
• a step for iteratively updating embeddings of the nodes and edges of the graph based on a spatiotemporal relationship between a feature in the target frame and the reference instance of the object in the reference frame to produce fused node embeddings and updated edge embeddings;
• a step for identifying a target instance of the object in the target frame based on the fused node embeddings;
• and
• displaying the target instance of the object as visually segmented from the target frame.