Selective adaptation of pre-trained neural network models

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining a pre-trained neural network model comprising a plurality of layers, the plurality of layers having a corresponding plurality of pre-trained weight matrices;
• augmenting at least one layer of the plurality of layers with a plurality of parameter-efficient adaptation modules, each parameter-efficient adaptation module of the plurality of parameter-efficient adaptation modules configured to generate a respective weight update matrix based on a low-rank factorization and associated with a respective learnable scoring parameter;
• fine-tuning the pre-trained neural network model on a target dataset to generate a fine-tuned model while the pre-trained weight matrices are maintained in a froze state, the fine-tuning comprising, for each of a plurality of training iterations:
• performing a forward pass, wherein for each parameter-efficient adaptation module, an indicator function selectively applies the respective weight update matrix based on a comparison of the respective learnable scoring parameter to a predetermined threshold;
• determining a total loss value based on a combination of a task-specific loss and a regularization term, the regularization term configured to induce sparsity by applying a penalty proportional to a norm of the respective learnable scoring parameters;
• and
• updating parameters of the plurality of parameter-efficient adaptation modules and the respective learnable scoring parameters based on the total loss value;
• and
• providing the fine-tuned model, comprising a sparse subset of activated parameter-efficient adaptation modules, for performing an inference task.

A computer-readable medium having instructions that, when executed by data processing hardware, causes the data processing hardware to perform operations comprising:

• obtaining a pre-trained neural network model comprising a plurality of layers, the plurality of layers having a corresponding plurality of pre-trained weight matrices;
• augmenting at least one layer of the plurality of layers with a plurality of parameter-efficient adaptation modules, each parameter-efficient adaptation module of the plurality of parameter-efficient adaptation modules configured to generate a respective weight update matrix based on a low-rank factorization and associated with a respective learnable scoring parameter;
• fine-tuning the pre-trained neural network model on a target dataset to generate a fine-tuned model while the pre-trained weight matrices are maintained in a froze state, the fine-tuning comprising, for each of a plurality of training iterations:
• performing a forward pass, wherein for each parameter-efficient adaptation module, an indicator function selectively applies the respective weight update matrix based on a comparison of the respective learnable scoring parameter to a predetermined threshold;
• determining a total loss value based on a combination of a task-specific loss and a regularization term, the regularization term configured to induce sparsity by applying a penalty proportional to a norm of the respective learnable scoring parameters;
• and
• updating parameters of the plurality of parameter-efficient adaptation modules and the respective learnable scoring parameters based on the total loss value;
• and
• providing the fine-tuned model, comprising a sparse subset of activated parameter-efficient adaptation modules, for performing an inference task.