Research Scientist @AIatMeta (FAIR) • PhD @coastalcph
Paris, France
Joined July 2020
- Tweets51
- Following605
- Followers395
- Likes907
Pinned Tweet
Happy to share our paper on language modelling with pixels has been accepted to ICLR‘23 (notable-top-5% / oral) 🎉. Big thanks and congrats to Team-PIXEL @jonasflotz @ebugliarello @esalesk @mdlhx @delliott and looking forward to presenting in Kigali! 🌍 #ICLR2023
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸
“Language Modelling with Pixels”
📄 arxiv.org/abs/2207.06991
🧑💻github.com/xplip/pixel
🤖huggingface.co/Team-PIXEL/pi…
by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott
Phillip Rust retweeted
(1/n) Excited to share our new paper from @AIatMeta FAIR: "EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data" 👶Human infants learn language from sparse, noisy multimodal input. Today's VLMs can't. We built a benchmark + challenge to close that gap.
w/ @rust_phillip, @angelvillar96, @juanmiguelpino, @mcxfrank, Emmanuel Dupoux and many other amazing collaborators. 🧵
Phillip Rust retweeted
Tough week! I also got impacted less than 3 months after joining. Ironically, I just landed some new RL infra features the day before.
Life moves on. My past work spans RL, PEFT, Quantization, and Multimodal LLMs.
If your team is working on these areas, I’d love to connect.
Phillip Rust retweeted
Humans see text — but LLMs don’t.
I wrote a short blog post exploring how models can perceive text visually rather than tokenize it:
🔗 csu-jpg.github.io/Blog/peopl…
From PIXEL, CLIPPO, VisInContext, VIST to DeepSeek-OCR, this is a quick story of how vision-centric modeling is changing how machines read,
and a reflection on some of our own small efforts in the past two years.
I will be presenting this work in-person at ACL🇹🇭 this week. Drop by if you'd like to chat!
Oral: Today (Monday) 16:30
Poster: Tuesday (Tomorrow) 10:30 - 12:00
Introducing “Towards Privacy-Aware Sign Language Translation at Scale”
We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data.
📄: arxiv.org/abs/2402.09611
🧵(1/9)
ALT This image shows the title and abstract of the paper linked in the tweet.
Introducing “Towards Privacy-Aware Sign Language Translation at Scale”
We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data.
📄: arxiv.org/abs/2402.09611
🧵(1/9)
ALT This image shows the title and abstract of the paper linked in the tweet.
For more experiments and all the details, check out our arXiv preprint linked above. We are working on releasing our code and data, so stay tuned! 👨💻
🧵(8/9)
This project is a collaboration with my amazing peers and mentors during my internship @AIatMeta: Bowen Shi, @skylrwang, @ncihancamgoz @j_maillard. ⭐
🧵(9/9)
Phillip Rust retweeted
New preprint "Improving Language Understanding from Screenshots" w/ @zwcolin @AdithyaNLP @danqi_chen.
We improve language understanding abilities of screenshot LMs, an emerging family of models that processes everything (including text) via visual inputs
arxiv.org/abs/2402.14073
Phillip Rust retweeted
In PHD: Pixel-Based Language Modeling of Historical Documents with @NadavBorenstein @rust_phillip and @IAugenstein, we apply pixel language models to processing historical document and to more standard NLP classification tasks too. See it in Poster Session 6 on Sunday 10th.
Phillip Rust retweeted
In Text Rendering Strategies for Pixel Language Models with @jonasflotz @rust_phillip and @esalesk, we design new text renderers for visual language processing to improve performance or to squeeze the model down to just 22M parameters. See it in Poster Session 2 on Friday 8th.
Phillip Rust retweeted
anon policy survey is out:
tinyurl.com/aclarxivpolicy
Phillip Rust retweeted
Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model.
This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task.
Details ⬇️
Phillip Rust retweeted
📢 I am hiring a postdoc to join our project on pixel-based natural language processing. The position is based in Copenhagen 🇩🇰 for 18 months. Applications are due by March 29 employment.ku.dk/faculty/?sh…. Informal inquiries are welcome.
Thrilled to receive a grant from @VILLUMFONDEN to carry out blue-skies research on tokenization-free NLP veluxfoundations.dk/en/about…
I will hire Ph.Ds and Postdocs to build up the group so feel free to reach out. We're starting off with a paper at #ICLR2023 openreview.net/forum?id=FkSp…
ALT A roof covered with Velux windows and solar panels against a beautiful blue sky.
Phillip Rust retweeted
Thrilled to receive a grant from @VILLUMFONDEN to carry out blue-skies research on tokenization-free NLP veluxfoundations.dk/en/about…
I will hire Ph.Ds and Postdocs to build up the group so feel free to reach out. We're starting off with a paper at #ICLR2023 openreview.net/forum?id=FkSp…
ALT A roof covered with Velux windows and solar panels against a beautiful blue sky.