Hybrid spatio-temporal neural models for video compression

A system comprising:

• communications circuitry configured to receive, at a client device, bitstreams of a plurality of model parameters, a sequence level representation, and a cross-resolution representation for reconstructing a sequence of frames, wherein the plurality of model parameters comprises neural radiance network parameters;
• and
• control circuitry configured to:
• decode the bitstreams of the plurality of model parameters, the sequence level representation, and the cross-resolution representation;
• generate, via a channel transformer, a combined representation based on the sequence level representation and the cross-resolution representation;
• adapt a neural network model based on the neural radiance network parameters;
• reconstruct the sequence of frames by determining, via the adapted neural network model based on the combined representation, pixel attribute information for each frame of the reconstructed sequence of frames;
• and
• generate, for display at the client device, the reconstructed sequence of frames, wherein the cross-resolution representation comprises latent features corresponding to each frame of the sequence of frames.

A method comprising:

• receiving, at a client device, bitstreams of a plurality of model parameters, a sequence level representation, and a cross-resolution representation for reconstructing a sequence of frames, wherein the plurality of model parameters comprises neural radiance network parameters;
• decoding the bitstreams of the plurality of model parameters, the sequence level representation, and the cross-resolution representation;
• generating, via a channel transformer, a combined representation based on the sequence level representation and the cross-resolution representation;
• adapting a neural network model based on the neural radiance network parameters;
• reconstructing the sequence of frames by determining, via the adapted neural network model based on the combined representation, pixel attribute information for each frame of the reconstructed sequence of frames;
• and
• generating, for display at the client device, the reconstructed sequence of frames,
• wherein the sequence of frames corresponds to a first resolution, the method further comprising using a convolutional network model comprising a plurality of residual spatial attention blocks to generate the combined representation at the first resolution.