Neural network systems for decomposing video data into layered representations

A method of training a neural network system that comprises a predictor 3D spatio-temporal encoder neural network and a predictor 3D spatio-temporal decoder neural network, comprising:

• generating a training data set comprising a plurality of training videos by, for each of the training videos, combining different first and second videos;
• and
• providing the training data set to the neural network system to generate output video data comprising two of sets of decomposed video data;
• and
• training the neural network system by adjusting parameters of the neural networks using a permutation invariant loss function, wherein the permutation invariant optimization function is dependent upon a reconstruction loss defining, for each training video, a difference between the sets of decomposed video data and the first and second videos and is invariant as to which of the first and second videos is reconstructed by which set of decomposed video data.

A method of processing input video data representing input video image frames to decompose the input video data into two or more sets of decomposed video data each representing a respective sequence of decomposed video image frames, the sequences of decomposed video image frames representing a decomposition of the input video image frames, the method comprising:

• receiving a sequence of input video image frames;
• encoding the sequence of input video image frames using a predictor 3D spatio-temporal encoder neural network, into a first set of latent variables representing a compressed version of the input video image frames;
• processing the first set of latent variables using a predictor 3D spatio-temporal decoder to generate two or more sets of decomposed video data representing respective sequences of decomposed video image frames;
• and
• providing output video data derived from one or more of the sets of decomposed video data.