Driving scenario understanding

A computer-implemented method for intersection scenario description, comprising:

• receiving a video stream of a surrounding environment of an ego-vehicle;
• extracting tracklets and appearance features associated with one or more dynamic objects from the surrounding environment;
• extracting motion features associated with one or more dynamic objects from the surrounding environment based on the corresponding tracklets;
• passing the appearance features through an appearance neural network to generate an appearance model;
• passing the motion features through a motion neural network to generate a motion model;
• passing the appearance model and the motion model through a fusion network to generate a fusion output;
• passing the fusion output through a classifier to generate a classifier output;
• and
• passing the classifier output through a loss function to generate a multi-label classification output indicative of the ego-vehicle, one or more of the dynamic objects, and one or more corresponding motion paths.

A system for intersection scenario description, comprising:

• a processor;
• and
• a memory storing instructions which, when executed by the processor cause the processor to perform:
• receiving a video stream of a surrounding environment of an ego-vehicle;
• extracting tracklets and appearance features associated with one or more dynamic objects from the surrounding environment;
• extracting motion features associated with one or more dynamic objects from the surrounding environment based on the corresponding tracklets;
• passing the appearance features through an appearance neural network to generate an appearance model;
• passing the motion features through a motion neural network to generate a motion model;
• passing the appearance model and the motion model through a fusion network to generate a fusion output;
• passing the fusion output through a classifier to generate a classifier output;
• and
• passing the classifier output through a loss function to generate a multi-label classification output indicative of the ego-vehicle, one or more of the dynamic objects, and one or more corresponding motion paths,
• wherein during weakly supervised phrase grounding, a last fully connected layer of the fusion network and a last fully connected layer of the classifier include a set of weights between a number of nodes from the appearance neural network and the motion neural network and a total number of classes from the loss function.