Fusion of motion and appearance features for object detection and trajectory prediction

An apparatus implementable in a vehicle, comprising:

• a processor capable of performing operations comprising:
• receiving an input stream of image frames;
• fusing a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames using a deep neural network (DNN) to generate an augmented stream of the image frames;
• and
• performing object detection and trajectory prediction of an object in the image frames based on the augmented stream,
• wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames, the processor fuses the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames with either of a late fusion architecture and a slow fusion architecture,
• wherein the late fusion architecture involves:
• processing the spatiotemporal input stream and the appearance-based stream through two separate stacks each comprising alternating convolutional layers and pooling layers to create two separate sets of feature maps;
• and
• combining the two sets of feature maps, and
• wherein the slow fusion architecture involves:
• processing each of the spatiotemporal input stream and the appearance-based stream individually through a convolutional layer and a pooling layer;
• and
• further processing the spatiotemporal input stream and the appearance-based stream together through a single stack comprising alternating convolutional layers and pooling layers.