Robust vision transformers
AssigneeNVIDIA Corporation
InventorsDaquan Zhou, Zhiding Yu, Enze Xie, Anima Anandkumar, Chaowei Xiao, Jose Manuel Alvarez Lopez
Life of this patent
Estimating the term…
Abstract
Apparatuses, systems, and techniques to generate a robust representation of an image. In at least one embodiment, input tokens of an input image are received, and an inference about the input image is generated based on a vision transformer (ViT) system comprising at least one self-attention module to perform token mixing and a channel self-attention module to perform channel processing.
Claims26
A system, comprising:
- at least one processor;
- at least one memory comprising instructions that, in response to execution by the at least one processor, cause the system to at least:
- receive one or more input tokens of an input at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion, wherein the one or more input tokens correspond to respective portions of the input;
- aggregate, at the at least one self-attention portion, the one or more input tokens;
- generate one or more feature representations of the input tokens based, at least in part, on the aggregation of the one or more input tokens;
- aggregate, at the at least one channel self-attention portion, the one or more feature representations of the input tokens, wherein the channel self-attention portion computes an attention matrix along a channel dimension of the input, wherein the one or more feature representations are aggregated based, at least in part, on a correlation value, between feature channels, that exceeds a threshold value;
- generate an inference about the input based, at least in part, on the aggregated one or more feature representations;
- and
- generate an output based, at least in part, on the inference.
A method, comprising:
- receiving one or more input tokens of an input image at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion;
- aggregating, by the at least one self-attention portion, the one or more input tokens;
- generating one or more feature representations of the input tokens based, at least in part, on the aggregated one or more input tokens;
- aggregating, by the at least one channel self-attention portion, the one or more feature representations of the input tokens, wherein the at least one channel self-attention portion computes an attention matrix along a channel dimension of the input image;
- generating an inference about the input image based, at least in part, on the aggregated one or more feature representations;
- and
- generating an output based, at least in part, on the inference.
A non-transitory computer-readable medium comprising instructions that, in response to execution by a computing device, cause the computing device to at least:
- receive a plurality of input tokens of an input image at one or more layers of a transformer, the one or more layers comprising at least one self-attention portion and at least one channel self-attention portion;
- aggregate, at the at least one self-attention portion, the one or more input tokens;
- generate one or more feature representations of the plurality of input tokens based, at least in part, on the aggregation of the plurality of input tokens;
- aggregate, at the at least one channel self-attention portion, the one or more feature representations;
- generate an inference about the input image based, at least in part, on the aggregated one or more feature representations;
- and
- generate an output based, at least in part, on the inference.
A method of operating a neural network to process an input image and from it generate an output image, the input image divisible into one or more patches and having a channel dimension, the method comprising:
- aggregating one or more feature representations of the one or more patches;
- aggregating the one or more feature representations, at least in part by computing an attention matrix along the channel dimension;
- generating an inference about the input image based, at least in part, on the aggregated one or more feature representations;
- and
- generating the output based, at least in part, on the inference.
Dependencies are inferred from the claim text.