Training-free architecture search for efficient vision transformers via approximated attention statistics

A processor-implemented method performed by at least one processor, the processor-implemented method comprising:

• generating a set of transformer model candidates for a target device, each transformer model candidate of the set of transformer model candidates being initialized with random weights;
• randomly sampling a set of data samples to produce random data samples for inputting at each transformer model candidate;
• computing an attention confidence score for each transformer model candidate based on the random data samples and the random weights;
• and
• selecting a transformer model candidate for the target device based on the attention confidence score.

A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor and comprising:

• program code to generate a set of transformer model candidates for a target device, each transformer model candidate of the set of transformer model candidates being initialized with random weights;
• program code to randomly sample a set of data samples to produce random data samples for inputting at each transformer model candidate;
• program code to compute an attention confidence score for each transformer model candidate based on the random data samples and the random weights;
• and
• program code to select a transformer model candidate for the target device based on the attention confidence score.

An apparatus comprising:

• means for generating a set of transformer model candidates for a target device, each transformer model candidate of the set of transformer model candidates being initialized with random weights;
• means for randomly sampling a set of data samples to produce random data samples for inputting at each transformer model candidate;
• means for computing an attention confidence score for each transformer model candidate based on the random data samples and the random weights;
• and
• means for selecting a transformer model candidate for the target device based on the attention confidence score.