Systems and methods for classifying a vehicle maneuver using a spatiotemporal attention selector

A device, comprising:

• one or more processors configured to:
• receive video data that includes a plurality of video frames and corresponding global positioning system (GPS) data and inertial measurement unit (IMU) data associated with a vehicle;
• process the video data, with an object detector model, to identify objects in the video data and to generate a first feature vector based on the objects,
• wherein the object detector model includes a faster region-based convolutional neural network model and a residual neural network model;
• process the GPS data and the IMU data, with a first convolutional neural network (CNN) model, to generate a second feature vector;
• process the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector based on the positions and the classes;
• utilize a second CNN model to generate a matrix of object features based on the first feature vector, the second feature vector, and the third feature vector;
• utilize a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle as safe or unsafe;
• and
• perform one or more actions based on the classification.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• receive video data that includes a plurality of video frames and corresponding global positioning system (GPS) data and inertial measurement unit (IMU) data associated with a vehicle;
• process the video data, with an object detector model, to identify objects in the video data and to generate a first feature vector based on the objects;
• process the GPS data and the IMU data, with a first convolutional neural network (CNN) model, to generate a second feature vector,
• wherein the first CNN model includes two-dimensional depth-wise separable convolutions;
• process the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector based on the positions and the classes;
• utilize a second CNN model to generate a matrix of object features based on the first feature vector, the second feature vector, and the third feature vector;
• utilize a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle as safe or unsafe;
• and
• perform one or more actions based on the classification.