GIFT: Generative Parameter-Efficient Fine-Tuning
Preprint
Chinmay Savadikar, Xi Song, Tianfu Wu
pdf web code

GIFT induces an explicit and direct mapping between the fine-tuned model and the frozen pretrained model, i.e., learns the fine-tuned weights directly from the pretrained weights. We show that the finetuned weights can be learned using a simple linear transformation of the pretrained weights, \(\hat{\omega}=\omega \cdot (\mathbb{I}+\phi_{d_{in}\times r}\cdot \psi_{r\times d_{in}})\), where trainable parameters \(\phi_{d_{in}\times r}\) and \( \psi_{r\times d_{in}}\) are shared across all the pretrained layers. GIFT outperforms prior methods on multiple Natural Language benchmarks using Llama series of models. On image classification tasks, the output of the first linear layer in GIFT plays the role of a \(r\)-way segmentation head without being explicitly trained to do so.

Continual Learning via Learning a Continual Memory in Vision Transformer
Preprint
Chinmay Savadikar, Michelle Dai, Tianfu Wu
pdf

We present a method to add lightweight, learnable parameters to Vision Transformers while leveraging parameter-heavy, but stable components. We show that the final linear projection layer in the multi-head self-attention (MHSA) block can be used as this light-weight module using a Mixture of Experts framework. While most of the prior methods which address this problem induce learnable parameters at every layer, or heuristically choose where to do so, we use Neural Architecture Search to determine this automatically. We use SPOS Neural Architecture Search and propose a task-similarity oriented sampling strategy to replace the uniform sampling and achieve better performance and efficiency than uniform sampling.