Transformer-based architecture for transform coding of media

An apparatus for processing media data, comprising:

• at least one memory;
• and
• at least one processor coupled to the at least one memory, the at least one processor configured to:
• segment a frame into a plurality of patches;
• and
• encode, based on a plurality of encoder transformer layers of an encoder sub-network using the plurality of patches as input, a frame to generate encoded image data, wherein, to encode the frame, the at least one processor is configured to:
• apply a first transformer, using a first transformer block of a first decoder transformer layer of the plurality of decoder transformer layers, to determine self-attention locally within one or more first window partitions applied over the one or more patches;
• apply a second transformer, using a second transformer block of the first encoder transformer layer, to determine self-attention locally within one or more shifted window partitions, wherein the one or more shifted window partitions overlap the one or more window partitions;
• determine, using one or more of the first transformer block and the second transformer block, one or more patches of features for applying a non-linear transform to the segmented frame;
• increase, using a patch merging engine, a patch size between the first encoder transformer layer and a second encoder transformer layer;
• and
• apply a non-linear transform to data comprising the one or more patches of features to compress the frame of encoded image data wherein the non-linear transform comprises a synthesis transform.

A method of processing media data, the method comprising:

• obtaining a latent representation of a frame of encoded image data;
• and
• generating, by a plurality of decoder transformer layers of a decoder sub-network using the latent representation of the frame of encoded image data as input, a frame of decoded image data, wherein at least one decoder transformer layer of the plurality of decoder transformer layers includes:
• one or more transformer blocks for generating one or more patches of features and determine self-attention locally within one or more window partitions and shifted window partitions applied over the one or more patches;
• and
• a patch un-merging engine for decreasing a respective size of each patch of the one or more patches, wherein generating, by the plurality of decoder transformer layers, the frame of decoded image data includes:
• applying a first transformer, by a first transformer block of a first decoder transformer layer of the plurality of decoder transformer layers, to determine self-attention locally within one or more first window partitions applied over the one or more patches;
• applying a first transformer, by a second transformer block of the first decoder transformer layer, to determine self-attention locally within one or more second window partitions applied over the one or more patches, wherein the one or more second window partitions are shifted to overlap one or more boundaries between adjacent ones of the one or more first window partitions;
• segmenting, by the patch un-merging engine, each patch of the one or more patches into a plurality of un-merged patches, wherein the plurality of un-merged patches are non-overlapping;
• and
• applying a non-linear transform to data comprising the one or more patches of features to generate a frame of decoded image data that is a reconstruction of the frame of encoded image data, wherein applying the non-linear transform comprises applying a synthesis transform and wherein the reconstruction of the frame of encoded image data comprises a reconstruction of an input image associated with the frame of encoded image data.

A method of processing media data, the method comprising:

• segmenting a frame into a plurality of patches;
• and
• encoding, by a plurality of encoder transformer layers of an encoder sub-network using the plurality of patches as input, a frame of to generate encoded image data, wherein encoding the frame comprises:
• applying a first transformer, by a first transformer block of a first encoder transformer layer of the plurality of encoder transformer layers, to determine self-attention locally within one or more window partitions;
• applying a second transformer, by a second transformer block of the first encoder transformer layer, to determine self-attention locally within one or more shifted window partitions, wherein the one or more shifted window partitions overlap the one or more window partitions;
• determining, by one or more of the first transformer block and the second transformer block, one or more patches of features for applying a non-linear transform to the segmented frame;
• increasing, by a patch merging engine, a patch size between the first encoder transformer layer and a second encoder transformer layer;
• and
• applying a non-linear transform to data comprising the one or more patches of features to generate the frame of encoded image data, wherein applying the non-linear transform comprises applying a synthesis transform.