Image object segmentation based on temporal information

A non-transitory computer-readable medium embodying program code for generating multiple masks for an object depicted in a video, the program code comprising instructions which, when executed by a processor, cause the processor to perform operations comprising:

• receiving (i) a video file including multiple frames in which the object is at least partially visible and (ii) a first mask corresponding to a first frame of the multiple frames, wherein the first mask indicates a first location of the object in the first frame;
• extracting, from the first frame, a location feature map indicating the first location of the object, by applying an initializer subnetwork to the first frame and the first mask, wherein the initializer subnetwork is trained to determine the first location of the object based on the first mask;
• extracting, from a second frame of the multiple frames, an image feature map indicating attributes of the second frame, by applying an encoder subnetwork to the second frame, wherein the encoder subnetwork is trained to determine the attributes of the second frame based on the second frame;
• extracting a difference feature map indicating a second location of the object in the second frame, by applying a convolutional long-short term memory (“LSTM”) subnetwork to the location feature map and to the image feature map, wherein the convolutional LSTM subnetwork is trained to determine the second location in the second frame based on (i) memory information indicated by the location feature map that is received as a hidden state by the convolutional LSTM subnetwork via a memory input and (ii) image information indicated by the image feature map that is received by the convolutional LSTM subnetwork via an additional input;
• and
• generating, based on the difference feature map, a second mask indicating the second location of the object in the second frame, by applying a decoder subnetwork to the difference feature map.

An object segmentation system for generating a group of masks for an object depicted in a video, the object segmentation system comprising:

• a memory device storing instructions which, when executed by a processor, implement a mask extraction subnetwork, the mask extraction subnetwork including an encoder subnetwork, a convolutional long-short term memory (“LSTM”) subnetwork, and a decoder subnetwork;
• a means for receiving (i) a video file including multiple frames in which the object is at least partially visible and (ii) a initializing mask, wherein the initializing mask indicates an initial location of the object in a first frame of the multiple frames;
• a means for extracting, from the first frame, a first image feature map indicating attributes of the first frame by applying the encoder subnetwork to the first frame and the initializing mask, wherein the encoder subnetwork is trained to determine the attributes based on the first frame and the initializing mask;
• a means for extracting a difference feature map indicating a first location of the object in an additional frame of the multiple frames, by applying the convolutional LSTM subnetwork to the first image feature map, wherein the convolutional LSTM subnetwork is trained to determine the first location of the object in the additional frame based on (i) memory information that is received as a hidden state by the convolutional LSTM subnetwork via a memory input and (ii) image information indicated by the first image feature map that is received by the convolutional LSTM subnetwork via an additional input;
• and
• a means for generating, based on the difference feature map, a first mask indicating the first location of the object in the additional frame, by applying the decoder subnetwork to the difference feature map.