Automated video segmentation

A method for automated video segmentation carried out by a computing system, the method comprising:

• receiving a sequence of video frames;
• analytically constructing a video graph from the sequence of video frames, the video graph comprising nodes connected by edges, wherein each node corresponds to a different one of the respective video frames, and each edge connects a different particular pair of nodes, and is associated with both a temporal distance between the respective video frames corresponding to the particular pair of nodes, and a respective similarity metric quantifying similarity between respective features of the respective video frames corresponding to the particular pair of nodes;
• applying an artificial neural network (ANN) to the video graph to compute both (i) predicted labels for the nodes of the video graph corresponding to predicted video-frame labels of the video frames, and (ii) predicted clusters of the nodes corresponding to predicted video segments of the sequence comprising respective sub-sequences of the video frames associated with respective contextually-related subject content, wherein the ANN has previously been trained to compute both predicted training labels for each training node of a training video graph constructed from a training video sequence of training video frames, and predicted training clusters of the training nodes corresponding to predicted membership among respective training video segments of the corresponding training video frames;
• further applying the ANN as trained to respectively determine a predicted segment class of each predicted video segment;
• and
• recording in memory and/or displaying information associating each video frame with one of the predicted video segments, and, for each respective predicted segment class, an identification of all predicted video segments belonging to the respective segment class.

A system for automated video segmentation, the system comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the system to carry out operations including:
• receiving a training sequence of training video frames comprising a plurality of respective training video segments, wherein the respective training video segments comprise non-overlapping respective sub-sequences of the training video frames, each of the respective sub-sequences being associated with respective contextually-related subject content, and wherein each training video frame is associated with a respective label identifying both which particular training video segment it belongs to, and a segment class associated with the particular training video segment;
• analytically constructing a training video graph from the training sequence of training video frames, the training video graph comprising nodes connected by edges, wherein each node corresponds to a different one of the respective training video frames, and each edge connects a different particular pair of nodes, and is associated with both a temporal distance between the respective training video frames corresponding to the particular pair of nodes, and a respective similarity metric quantifying similarity between respective features of the respective training video frames corresponding to the particular pair of nodes;
• training an artificial neural network (ANN) to compute both (i) a predicted training label for each node of the training video graph, and (ii) predicted clusters of the nodes corresponding to predicted membership among the respective training video segments of the corresponding training video frames, using the training video graph as input to the ANN, and ground-truth clusters of ground-truth labeled nodes, wherein the ground-truth labeled nodes correspond to the training video frames and their associated respective labels, and the ground-truth clusters correspond to the respective training video segments;
• further training the ANN to compute a predicted segment class for each of the predicted clusters, using as ground truths the segment classes of the respective training video segments;
• and
• configuring the trained ANN for application to one or more unseen runtime video sequences.

The system of 19, wherein analytically constructing the runtime video graph from the runtime sequence of runtime video frames comprises:

• extracting from each respective runtime video frame timing information indicating temporal position in the runtime sequence, and a respective runtime feature vector characterizing media data of the respective runtime video frame;
• creating a respective runtime node associated with each respective runtime video frame and its respective runtime feature vector and timing information;
• and
• for every respective pair of created runtime nodes, determining a connecting edge having a length corresponding to a temporal distance between the pair of runtime video frames associated with respective pair of runtime nodes, and a weight corresponding to a similarity metric of the respective runtime feature vectors of the pair of runtime video frames associated with the respective pair of runtime nodes,
• wherein applying the trained ANN to the runtime video graph to compute both (i) the predicted runtime label for each runtime node of the runtime video graph and (ii) the predicted runtime clusters of the runtime nodes comprises:
• applying the trained GNN to the runtime video graph to predict a respective embedding runtime vector for each respective runtime node of the runtime video graph;
• and
• applying the trained clustering ANN to the respective embedding runtime vectors to predict clusters of the runtime nodes, and to predict node labels corresponding to the respective runtime labels of the runtime video frames,
• and wherein further applying the trained ANN to respectively determine both the predicted runtime video segment associated with each predicted runtime cluster, and the predicted runtime segment class of each predicted runtime video segment comprises:
• applying the trained clustering ANN to both the predicted clusters of the runtime nodes and the predicted node labels in order to predict the runtime video segment associated with each predicted runtime cluster, and to predict the runtime segment class of each predicted runtime video segment.