rodolphe_jenatton retweeted
Introducing Soft MoE! Sparse MoEs are a popular method for increasing the model size without increasing its cost, but they come with several issues. Soft MoEs avoid them and significantly outperform ViT and different Sparse MoEs on image classification.
arxiv.org/abs/2308.00951
rodolphe_jenatton retweeted
New paper from my time as student researcher at Google :)
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
introduce Three Towers (3T), a flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers. While contrastive models are usually trained from scratch, LiT (Zhai et al., 2022) has recently shown performance gains from using pretrained classifier embeddings. However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits of contrastively training the image tower. With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training. To achieve this, we introduce a third tower that contains the frozen pretrained embeddings, and we encourage alignment between this third tower and the main image-text towers. Empirically, 3T consistently improves over LiT and the CLIP-style from-scratch baseline for retrieval tasks. For classification, 3T reliably improves over the from-scratch baseline, and while it underperforms relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k and Places365 pretraining.
paper page: huggingface.co/papers/2305.1…
rodolphe_jenatton retweeted
Excited & proud to share our work on Scaling Vision Transformers to 22B params, i.e. largest vision model to date🚀!
arxiv.org/abs/2302.05442
ViT-22B achieves excellent transfer on dense recognition tasks, i.e. semantic segmentation & depth prediction, with a *frozen* backbone❄️
rodolphe_jenatton retweeted
A large team effort from Brain EMEA and collaborators to train and evaluate a large Vision Transformer.
Many of us here believe that vision can be scaled successfully as LLMs have, and are making steps in that direction!
1/ There is a huge headroom for improving capabilities of our vision models and given the lessons we've learned from LLMs, scaling is a promising bet. We are introducing ViT-22B, the largest vision backbone reported to date: arxiv.org/abs/2302.05442
rodolphe_jenatton retweeted
We distilled extremely performant VIT B/16 & L/16 models from VIT-22B. They have just 0.4% and 1.4% the number of parameters VIT-22B contains, and improve the state of the art ImageNet accuracy at their sizes by a wide margin; see table below.
rodolphe_jenatton retweeted
Today we announce the largest vision model to date, VIT-22B. Larger models do better, while training & inference costs increase accordingly. This may rule out their direct use in practical scenarios. Fortunately, we show smaller models also benefit from large foundation ones.
rodolphe_jenatton retweeted
2️⃣2️⃣🅱️: We trained a 22B parameter ViT model, and scale continues to prove its merit! I want to zero in on an aspect of this which is useful however at all scales: a method for improving training stability in transformers.
arxiv.org/abs/2302.05442
rodolphe_jenatton retweeted
Delighted to share something I was focusing on heavily - Scaling Vision Transformers to 22 billion parameters
arxiv.org/abs/2302.05442
ViT-22B brings best of architectural choices and recipes from previous language and vision transformers for an encoder only model. (1/5)
ALT Screenshot of the front page of "Scaling Vision Transformers to 22 Billion Parameters" paper.
rodolphe_jenatton retweeted
1/ There is a huge headroom for improving capabilities of our vision models and given the lessons we've learned from LLMs, scaling is a promising bet. We are introducing ViT-22B, the largest vision backbone reported to date: arxiv.org/abs/2302.05442
rodolphe_jenatton retweeted
Introducing Vit-22B!
Vit-22B is the largest dense Vision transformer ever trained 🔥.
It's time for vision to catch up to language in the scaling game!
I am excited to find out what other emergent abilities can be found by scaling up vision models!
1/ There is a huge headroom for improving capabilities of our vision models and given the lessons we've learned from LLMs, scaling is a promising bet. We are introducing ViT-22B, the largest vision backbone reported to date: arxiv.org/abs/2302.05442
rodolphe_jenatton retweeted
1/ In 2021, we shared next-gen language + conversation capabilities powered by our Language Model for Dialogue Applications (LaMDA). Coming soon: Bard, a new experimental conversational #GoogleAI service powered by LaMDA.
blog.google/technology/ai/ba…
rodolphe_jenatton retweeted
The recordings from the Google Workshop on Sparsity and Adaptive Computation are now available at:
Day 1: youtube.com/watch?v=QBqRnQ_1… (includes the keynotes from Dave Patterson and Jeff Dean)
Day 2: youtube.com/watch?v=I39Rl7AX…
The agenda for the talks covered in the videos is below.
ALT Agenda for the Google Workshop on Sparsity and Adaptive Computation.
rodolphe_jenatton retweeted
SAVAGE!
🐐 @EliudKipchoge
𝑾𝑶𝑶𝑶𝑶𝑶𝑶𝑶𝑹𝑳𝑫 𝑹𝑬𝑪𝑶𝑹𝑫!!!! 🐐
Eliud Kipchoge wins the @berlinmarathon in a new World Record of 2:01:09!! 🤯
#NNRunningTeam
rodolphe_jenatton retweeted
Great collaboration between Deepmind and the Brain team at Google Research using transformers to meta learn hyperparameter optimization.
Today on the blog learn about the OptFormer, one of the first Transformer-based frameworks for hyperparameter tuning, learned from large-scale optimization data using flexible text-based representations → goo.gle/3we0HvJ
rodolphe_jenatton retweeted
Presenting Plex's poster today with a few folks at the Pretraining ICML workshop. Drop by if you'd like to hear about it or chat.
Arrived at #ICML2022. Please say hi if you see me around! I'll be giving:
1. A talk on Uncertainty in ML at the Astrophysics workshop Friday ml4astro.github.io/icml2022
2. A talk on Plex at the Pretraining workshop Saturday pretraining.github.io
rodolphe_jenatton retweeted
Expert modules let the model encapsulate useful specialized knowledge (see plot!). Text-only, image-only and text-and-image experts organically emerge! arxiv.org/abs/2206.02770
joint work w/ awesome @_basilM @joapuipe @RJenatton @neilhoulsby
rodolphe_jenatton retweeted
Beep beep! Introducing LIMoE, the Language Image Mixture of Experts: a single model, processing both modalities for contrastive image-text modelling. Cruises straight to 84.1% 0shot ImageNet accuracy without any modality-specific architectures or pre-training. (1/10)
rodolphe_jenatton retweeted
I'm looking for PhD students and Post-docs to work in the area of optimization for machine learning, and the theory of deep learning.
Please retweet!
rodolphe_jenatton retweeted
Vision Transformers can drop many tokens (say, background, redundant, noise, etc) & still succeed at image classification. We propose an extremely simple way to merge tokens and save up to 40-50% training cost in huge models while keeping their performance arxiv.org/abs/2202.12015
rodolphe_jenatton retweeted
Excited to host our next speaker, Sebastian Nowozin, next Thursday 3rd March , at 10am EST / 4pm CET, to talk about "I Can’t Believe Bayesian Deep Learning is not Better" (abstract and registration details at icbinb.cc/seminar-series/)
