Robust vision transformers
AssigneeNVIDIA Corporation
InventorsDaquan Zhou, Zhiding Yu, Enze Xie, Anima Anandkumar, Chaowei Xiao, Jose Manuel Alvarez Lopez
Life of this patent
Estimating the term…
Abstract
Apparatuses, systems, and techniques to generate a robust representation of an image. In at least one embodiment, input tokens of an input image are received, and an inference about the input image is generated based on a vision transformer (ViT) system comprising at least one self-attention module to perform token mixing and a channel self-attention module to perform channel processing.
Claims20
A method of operating a neural network to process an input image and from the input image generate an output image, the input image divisible into one or more patches and having a channel dimension, the method comprising:
- aggregating one or more feature representations of the one or more patches to obtain one or more aggregated feature representations;
- aggregating the one or more aggregated feature representations, at least in part by computing an attention matrix along the channel dimension to obtain one or more channel-aggregated feature representations;
- generating an inference about the input image based, at least in part, on the one or more channel-aggregated feature representations;
- and
- generating the output image based, at least in part, on the inference.
A system comprising:
- one or more processors;
- at least one memory comprising instructions that, in response to execution by the one or more processors, cause the system to operate one or more neural networks to at least:
- aggregate one or more token embeddings encoding one or more patches of an input image to obtain one or more token-aggregated feature representations;
- compute an attention matrix along a channel dimension of the input image;
- use the attention matrix to aggregate the one or more token-aggregated feature representations to obtain one or more channel-aggregated feature representations;
- generate an inference about the input image based, at least in part, on the one or more channel-aggregated feature representations;
- and
- generate output based, at least in part, on the inference.
One or more processors comprising:
- circuitry to:
- aggregate one or more embeddings associated with one or more patches of an input image to obtain one or more first feature representations;
- aggregate the one or more first feature representations, at least in part by computing an attention matrix along a channel dimension of the input image to obtain one or more second feature representations;
- generate an inference about the input image based, at least in part, on the one or more second feature representations;
- and
- generate output based, at least in part, on the inference.
Dependencies are inferred from the claim text.