Scene-aware video encoder system and method

A scene-aware video encoder system, comprising:

• at least one processor;
• and a memory having instructions stored thereon that, when executed by the at least one processor, cause the scene-aware video encoder system to:
• receive a sequence of video frames of a video of a scene;
• transform the sequence of video frames into a spatio-temporal scene graph including nodes representing one or multiple static objects and one or multiple dynamic objects in the scene, wherein an appearance and a location of each of the static objects in the scene are represented by properties of a single node of the spatio-temporal scene graph, and wherein each of the dynamic objects in the scene is represented by properties of multiple nodes of the spatio-temporal scene graph describing an appearance, a location, and a motion of each of the dynamic objects at different instances of time;
• encode the nodes of the spatio-temporal scene graph into a latent space using a spatio-temporal transformer encoding different combinations of different nodes of the spatio-temporal scene graph corresponding to different spatio-temporal volumes of the scene, wherein encoding of each node of the different nodes in each of the combinations is weighted with an attention score determined as a function of similarities of spatio-temporal locations of the different nodes in the combination;
• and
• output the encoded nodes of the spatio-temporal scene graph.