Back to results
US 12,682,626 B2Granted
Robust vision transformers
AssigneeNVIDIA Corporation
InventorsDaquan Zhou, Zhiding Yu, Enze Xie, Anima Anandkumar, Chaowei Xiao, Jose Manuel Alvarez Lopez
Life of this patent
Estimating the term…
Abstract
Apparatuses, systems, and techniques to generate a robust representation of an image. In at least one embodiment, input tokens of an input image are received, and an inference about the input image is generated based on a vision transformer (ViT) system comprising at least one self-attention module to perform token mixing and a channel self-attention module to perform channel processing.
Claims20
A system, comprising:
- at least one processor;
- at least one memory comprising instructions that, in response to execution by the at least one processor, cause the system to at least:
- receive one or more input tokens of an input at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion, wherein the one or more input tokens correspond to respective portions of the input;
- aggregate, at the at least one self-attention portion, the one or more input tokens;
- generate one or more feature representations of the input tokens based, at least in part, on the aggregation of the one or more input tokens;
- aggregate, at the at least one channel self-attention portion, the one or more feature representations of the input tokens, wherein the channel self-attention portion computes an attention matrix along a channel dimension of the input, wherein the one or more feature representations are aggregated based, at least in part, on a correlation value, between feature channels, that exceeds a threshold value;
- generate an inference about the input based, at least in part, on the aggregated one or more feature representations;
- and
- generate an output based, at least in part, on the inference.
6 more elements
A method, comprising:
- receiving one or more input tokens of an input image at one or more encoders, the one or more encoders comprising at least one self-attention portion and at least one channel self-attention portion;
- aggregating, by the at least one self-attention portion, the one or more input tokens;
- generating one or more feature representations of the input tokens based, at least in part, on the aggregated one or more input tokens;
- aggregating, by the at least one channel self-attention portion, the one or more feature representations of the input tokens by combining the one or more feature representations of the input tokens with an attention matrix indicating an amount of correlation between different channels associated with the input image;
- generating an inference about the input image based, at least in part, on the aggregated one or more feature representations;
- and
- generating an output based, at least in part, on the inference.
4 more elements
A non-transitory computer-readable medium comprising instructions that, in response to execution by a computing device, cause the computing device to at least:
- receive a plurality of input tokens of an input image at one or more layers of a transformer, the one or more layers comprising at least one self-attention portion and at least one channel self-attention portion;
- aggregate, at the at least one self-attention portion, the plurality of input tokens;
- generate one or more feature representations of the plurality of input tokens based, at least in part, on the aggregation of the plurality of input tokens;
- aggregate, at the at least one channel self-attention portion, the one or more feature representations using an attention matrix indicating an amount of correlation between different channels associated with the input image;
- generate an inference about the input image based, at least in part, on the aggregated one or more feature representations;
- and
- generate an output based, at least in part, on the inference.
4 more elements
Dependencies are inferred from the claim text.