Unsupervised learning of object representations from video sequences using attention over space and time

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a video object representation neural network system, the video object representation neural network system including a self-attention neural network comprising one or more self-attention layers, each configured to apply a self-attention mechanism over a layer input, and having a plurality of self-attention neural network parameters, the operations comprising:

• obtaining a video sequence comprising a sequence of T image frames, one at each of a succession of times;
• processing each of the image frames to generate a set of S spatial encodings for each of the T image frames thereby to obtain a group of S×T spatial encodings for the video sequence, wherein each spatial encoding of an image frame comprises an encoding of a respective region of the image frame;
• processing the group of S×T spatial encodings using the self-attention neural network by applying the self-attention mechanism over a layer input comprising the group of S×T spatial encodings to generate a group of transformed spatial encodings each corresponding to an image frame time and to an image frame region, wherein applying the self-attention mechanism over the layer input includes applying the self-attention mechanism over the spatial encodings of different image frames;
• determining, for each of a set of object latent variables, one or more values parameterizing an object latent distribution of the object latent variable, by aggregating the group of transformed spatial encodings over times of the image frames;
• determining a value for each of the set of object latent variables by sampling from the object latent distribution for the object latent variable;
• processing the determined values of the object latent variables to generate a decoded sequence of image frames;
• and
• training the system by adjusting at least values of the self-attention neural network parameters to optimize an objective function dependent on i) a measure of a difference between the sequence of T image frames and the decoded sequence of image frames and ii) a difference between each of the object latent distributions and a prior object latent distribution.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a video object representation neural network system, the video object representation neural network system including a self-attention neural network comprising one or more self-attention layers, each configured to apply a self-attention mechanism over a layer input, and having a plurality of self-attention neural network parameters, the operations comprising:

• obtaining a video sequence comprising a sequence of T image frames, one at each of a succession of times;
• processing each of the image frames to generate a set of S spatial encodings for each of the T image frames thereby to obtain a group of S×T spatial encodings for the video sequence, wherein each spatial encoding of an image frame comprises an encoding of a respective region of the image frame;
• processing the group of S×T spatial encodings using the self-attention neural network by applying the self-attention mechanism over a layer input comprising the group of S×T spatial encodings to generate a group of transformed spatial encodings each corresponding to an image frame time and to an image frame region, wherein applying the self-attention mechanism over the layer input includes applying the self-attention mechanism over the spatial encodings of different image frames;
• determining, for each of a set of object latent variables, one or more values parameterizing an object latent distribution of the object latent variable, by aggregating the group of transformed spatial encodings over times of the image frames;
• determining a value for each of the set of object latent variables by sampling from the object latent distribution for the object latent variable;
• processing the determined values of the object latent variables to generate a decoded sequence of image frames;
• and
• training the system by adjusting at least values of the self-attention neural network parameters to optimize an objective function dependent on i) a measure of a difference between the sequence of T image frames and the decoded sequence of image frames and ii) a difference between each of the object latent distributions and a prior object latent distribution.