System and method for forecasting location of target in monocular first person view

A system for forecasting location of a target in monocular first person view, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• receive, a sequence of past bounding boxes, wherein each bounding box of the sequence of past bounding boxes encloses the target in a frame from amongst a sequence of frames, and wherein the sequence of frames associated with a set of past locations of the target;
• predict, in real-time, a sequence of future bounding boxes corresponding to future locations of the target based on the sequence of past bounding boxes using a network model, wherein the network model comprising an encoder block having a first LSTM and a second LSTM operating collectively as an auto-encoder, a decoder block comprising a third LSTM and a trajectory concatenation layer, wherein to predict the sequence of future bounding boxes, the one or more hardware processors are further configured by the instructions to:
• determine, by the encoder block, a representation vector of predetermined size based on a bounding box information associated with the sequence of past bounding boxes, the bounding box information indicative of a history associated with the set of past locations of the target, wherein the bounding box information comprises a set of vectors associated with the sequence of past bounding boxes, a vector from amongst the set of vectors associated with a bounding box from amongst the sequence of past bounding boxes comprising a centroid, width, height, velocity of the centroid, and a change in the width and the height of the bounding box, wherein to determine the representation vector by the encoder block, the one or more hardware processors are further configured by the instructions to:
• generate, by the first LSTM, a final hidden state vector which summarizes a complete sequence of bounding box information;
• map, by a fully connected layer associated with the encoder block, the final hidden state vector to the representation vector of predetermined length via a ReLU;
• generate, by the second LSTM, a set of hidden state vectors in a plurality of iterations, wherein for each iteration, the second LSTM takes the representation vector as input;
• and
• pass through a fully connected layer associated with the second LSTM, the set of hidden state vectors generated in each iteration of the plurality of iterations;
• predict, by the decoder block, a future velocity and change in dimension of future bounding boxes of the target based on the representation vector, wherein predicting the future velocity and the change in dimension of future bounding boxes by the decoder block, the one or more hardware processors are further configured by the instructions to:
• receive the vector representation from the encoder block;
• generate, by the third LSTM, a set of hidden state vectors in a plurality of iterations, wherein to generate a hidden state vector for a current iteration of the plurality of iterations by the third LSTM, the one or more hardware processors are further configured by the instructions to:
• take, as input, the representation vector and hidden state vectors associated with iterations previous to the current iteration in the plurality of iterations;
• generate a hidden state vector from amongst the set of hidden state vectors based on the input;
• and
• map the hidden state vector to a vector of four dimensions indicative of velocity and dimension change components via a ReLU followed by a fully connected layer;
• and
• convert, by a trajectory concatenation layer, the future velocities and change in dimensions of future bounding boxes into the sequence of future bounding box of the target, wherein converting future velocities and change in dimensions of the future bounding boxes comprises converting the predicted future velocities of the centroids and the change in the dimension into a sequence of locations and dimension of the sequence of the future bounding boxes using the past bounding box locations.

One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving a sequence of past bounding boxes, wherein each bounding box of the sequence of past bounding boxes enclosing the target in a frame from amongst a sequence of frames, wherein the sequence of frames associated with a set of past locations of the target;
• predicting in real-time, a sequence of future bounding boxes corresponding to future locations of the target based on the sequence of past bounding boxes using a network model, wherein the network model comprising an encoder block having a first Long Short Term Memory (LSTM) and a second LSTM operating collectively as an auto-encoder, a decoder block comprising a third LSTM and a trajectory concatenation layer, wherein predicting the sequence of future bounding boxes comprises:
• determining, by the encoder block, a representation vector of a predetermined size based on a bounding box information associated with the sequence of past bounding boxes, the bounding box information indicative of a history associated with the set of past locations of the target, wherein the bounding box information comprises a set of vectors associated with the sequence of past bounding boxes, a vector from amongst the set of vectors associated with a bounding box from amongst the sequence of past bounding boxes comprising a centroid, width, height, velocity of the centroid, and a change in the width and the height of the bounding box, wherein determining the representation vector by the encoder block comprises:
• generating, by the first LSTM, a final hidden state vector which summarizes a complete sequence of bounding box information;
• mapping, by a fully connected layer associated with the encoder block, the final hidden state vector to the representation vector of predetermined length via a ReLU:
• generating, by the second LSTM, a set of hidden state vectors in a plurality of iterations, wherein for each iteration, the second LSTM takes the representation vector as input;
• and
• passing through a fully connected layer associated with the second LSTM, the set of hidden state vectors generated in each iteration of the plurality of iterations;
• predicting, by the decoder block, a future velocity and change in dimension of future bounding boxes of the target based on the representation vector, wherein predicting the future velocity and the change in dimension of future bounding boxes by the decoder block comprises:
• receiving the vector representation from the encoder block;
• generating, by the third LSTM, a set of hidden state vectors in a plurality of iterations, wherein generating a hidden state vector for a current iteration of the plurality of iterations by the third LSTM comprises:
• taking, as input, the representation vector and hidden state vectors associated with iterations previous to the current iteration in the plurality of iterations;
• generating a hidden state vector from amongst the set of hidden state vectors based on the input;
• and
• mapping the hidden state vector to a vector of four dimensions indicative of velocity and dimension change components via a ReLU followed by a fully connected layer;
• and
• converting, by a trajectory concatenation layer, the future velocities and change in dimensions of future bounding boxes into the sequence of future bounding box of the target, wherein converting future velocities and change in dimensions of the future bounding boxes comprises converting the predicted future velocities of the centroids and the change in the dimension into a sequence of locations and dimension of the sequence of the future bounding boxes using the past bounding box locations.