Senior Research Engineer at Google Deep Mind working on parameter-efficient adaptation and few-shot generalization, mostly within NLP. View are my own. he/him
Joined July 2013
- Tweets93
- Following243
- Followers447
- Likes94
Is Kevin onto something? We found that LLMs can struggle to understand compressed text, unless you do some specific tricks. Check out arxiv.org/abs/2404.03626 and help @hoonkp, @alemi, Jeffrey Pennington, @ada_rob, @jaschasd, @noahconst and I make Kevin’s dream a reality.
We just pushed a new update adding support for the (very impressive) safetensors library from our friends at @huggingface!
Git-Theta's plug-in system meant that we spent more time waiting on CI/CD than actually adding support (I'll get off my soapbox now 🧼📦).
Introducing Git-Theta, a Git extension that enables collaborative and continual development of ML models with merges, diffs, and parameter-efficient updates—all using the standard Git workflow!
📄 arxiv.org/abs/2306.04529
💽 github.com/r-three/git-theta
🗣️ cccml.zulipchat.com
🧵⬇️
Introducing Git-Theta, a Git extension that enables collaborative and continual development of ML models with merges, diffs, and parameter-efficient updates—all using the standard Git workflow!
📄 arxiv.org/abs/2306.04529
💽 github.com/r-three/git-theta
🗣️ cccml.zulipchat.com
🧵⬇️
Git-Theta is designed around plug-ins—this means that if we don’t support your favorite framework, merging strategy, or parameter-efficient update yet, you can add it! Join us on GitHub github.com/r-three/git-theta or Zulip cccml.zulipchat.com to start contributing!
This was joint work will wonderful collaborators: @kandpal_nikhil @Muqeeth10 @anisham197 @montymevans Vishal Baskaran @TenghaoHuang45 @liu_haokun and @colinraffel
.@MotiveStudio, I saw @PyTorch in the licenses for @deadspace. Are you using it as a GPU-accelerated linear algebra library or are there actually neural nets running during the game? #deadspace #deadspaceremake
Brian Lester retweeted
While parameter-efficient tuning methods are originally proposed to reduce computation & storage costs, it turns out they can help overcome catastrophic forgetting and thus improve performance on zero-shot cross-lingual generation. Checkout our work @GoogleAI @emnlpmeeting👇1/10
Am I missing something wrt to the name "gradient checkpointing"? Clearing cached activations and recomputing them in the backwards pass seems like the opposite of checkpointing. The name makes it sound like we are storing the activations on disk.
docs.aws.amazon.com/sagemake…
Brian Lester retweeted
We are presenting SPoT: Better Frozen Model Adaption through Soft Prompt Transfer @aclmeeting today during the 2pm in-person ML for NLP poster session and tomorrow at the 7:30am virtual poster session (virtual session w/@tuvuumass). #acl2022 #NLProc #ACLinDublin #acl2022nlp
Brian Lester retweeted
Happy to share our soft prompt transfer (SPoT) paper made it to #ACL2022 🎉.
On the SuperGLUE leaderboard, SPoT is the first parameter-efficient approach that is competitive with methods that tune billions of parameters.
w/ @blester125, @noahconst, @aboSamoor, @daniel_m_cer
Sharing my internship work @GoogleAI: 1) w/ Soft Prompt Transfer, Prompt Tuning matches or significantly outperforms Model Tuning across model sizes, 2) tasks can help each other via their prompts & task prompts can be used as task embeddings to formalize task similarity.
🧵 1/8
The blog post for my EMNLP 2021 paper on Prompt Tuning is out! Writing for a blog is pretty different than writing for a conference, so if anything was confusing in the paper maybe this will help it click (or you could have just asked me lol)
Fine-tuning pre-trained models is common in NLP, but forking the model for each task can be a burden. Prompt tuning adds a small set of learnable vectors to the input and can match fine-tuning quality while sharing the same frozen model across all tasks. goo.gle/3Bch2lL
ALT Prompt tuning combines the best features of model tuning and prompt design to enable pre-trained models to achieve strong task performance while being served efficiently in a multi-task setting. Like prompt design, the pre-trained model is fixed and shared across tasks. However, learned soft prompts are added to the model input, enabling stronger adaptation to specific tasks.
It took a bit, but, like the best desserts, it needed to cool before we could bite in. Our code for Prompt Tuning has been open sourced! It enables training with all T5 sizes on TPU, reproducing our results, and is a great starting point for YOUR work. github.com/google-research/p…
Huge thanks to my collaborators, the people who have put this library through its paces, and the T5X and Flaxformer authors! @noahconst @aboSamoor @tuvuumass @daniel_m_cer @GreenBeanDou @ada_rob @hwchung27 @anselmlevskaya and so many more.
In addition to the impressive performance gains, I'm incredibly excited about how this work opens new exploration of targeted transfer learning via prompt similarity. I can't wait to see what gets built on this!
Sharing my internship work @GoogleAI: 1) w/ Soft Prompt Transfer, Prompt Tuning matches or significantly outperforms Model Tuning across model sizes, 2) tasks can help each other via their prompts & task prompts can be used as task embeddings to formalize task similarity.
🧵 1/8
My first @GoogleAI residency project was accepted to @emnlpmeeting #EMNLP2021!
Prompt Tuning can condition a frozen T5 XXL model to perform new tasks while only adding 0.003% more parameters and no performance loss.
Camera Ready 📸: arxiv.org/abs/2104.08691
Quick Thread 🧵(1/7)
ALT A graph plotting model performance on SuperGLUE verse the number of parameters in the model. We multiple variations of Model Tuning are ahead of Prompt Tuning, but it closes the gap as we reach T5 XXL sized models.
An interesting quirk of Prompt Tuning is that the hyper parameters are a little strange. For example, our learning rate is 0.3, two orders of magnitude larger than the default T5 learning rate of 0.001. (6/7)
A huge shout out to my amazing mentors, @noahconst and @aboSamoor, who were a big part of making this project possible. (7/7)