Sub-tile-based grid sampling in neural video codecs

An integrated circuit, comprising:

• a memory;
• a processor coupled to the memory, the processor to determine a super tile of a video frame of a video, split the super tile into one or more sub-tiles, and determine a reference bounding box for a sub-tile of the one or more sub-tiles and per-pixel metadata for the sub-tile;
• a data movement engine to load a subset of reference data for the sub-tile according to the reference bounding box onto the memory;
• and
• a compute array coupled to the memory, the compute array to perform one or more operations using the subset of reference data loaded onto the memory and the per-pixel metadata and output data for the sub-tile based on the one or more operations.

A compute engine, comprising:

• an instruction decoder to decode one or more instructions, comprising an instruction to split a super tile of a video frame into one or more sub-tiles based on data corresponding to the super tile, the instruction comprising:
• one or more inputs including one or more memory addresses pointing to the data corresponding to the super tile, one or more size dimensions of the data corresponding to the super tile, and a criterion;
• and
• one or more outputs including a memory address for writing information specifying the one or more sub-tiles, and a memory address for writing metadata corresponding to the super tile and the one or more sub-tiles;
• and
• circuitry configurable by the instruction decoder to perform one or more operations corresponding to the instruction, the one or more operations comprise:
• splitting the super tile into one or more sub-tiles based on the data corresponding to the super tile and the criterion and calculating the metadata corresponding to the super tile and the one or more sub-tiles.