Methods and apparatus to perform dense prediction using transformer blocks

An apparatus comprising:

• a memory;
• instructions that when executed cause at least one processor to:
• convert an input image to a plurality (N) of tokens, respective ones of the N tokens based on respective non-overlapping patches of the input image, the N tokens to include positional information, the positional information to identify respective positions in which the respective non-overlapping patches fit within the input image;
• reassemble the N tokens into feature representations after the tokens have passed through transformer stages, ones of the tokens output by deeper ones of the transformer stages assembled at a first resolution, ones of the tokens output by early ones of the transformer stages assembled at a second resolution, the first resolution lower than the second resolution;
• progressively fuse the feature representations using consecutive stages of a residual network, and, in each stage of the residual network, upsample a respective representation output by a respective stage of the residual network by a factor of two;
• and
• generate a dense prediction based on the fused feature maps.

A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to at least:

• convert an input image into tokens, the tokens to represent features extracted from the input image;
• and
• transform the tokens with information relating each token to all the other tokens;
• reassemble the transformed tokens into feature representations;
• progressively fuse the feature representations to a generate a final feature representation,
• progressively upsample the final feature representation by a factor of two;
• and
• generate a dense prediction based on the final feature representation.

A method comprising:

• converting, by executing an instruction with at least one processor, an input image into tokens, the tokens to represent features extracted from the input image;
• and
• transforming, by executing an instruction with the at least one processor, the tokens to include information relating each token to all the other tokens;
• reassembling, by executing an instruction with the at least one processor, the transformed tokens into feature representations, the transformed tokens output from respective ones of a plurality of transformer stages;
• progressively fusing, by executing an instruction with the at least one processor, the feature representations to a generate a final feature representation,
• progressively, by executing an instruction with the at least one processor, upsampling the feature representations by a factor of two;
• and
• generating, by executing an instruction with the at least one processor, a dense prediction based on the final feature representation.