🌱 // prev @googledeepmind @ucirvine
little worm in big apple
Joined May 2017
- Tweets192
- Following514
- Followers476
- Likes5.8K
Pinned Tweet
Lots of discourse around long-context language models (LCLMs) subsuming RAG and retrieval but how close are we to this paradigm shift?
Introducing LOFT a 1 million token benchmark spanning 6 tasks & 35 datasets to test LCLMs’ ability to do in-context retrieval & reasoning [1/10]
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
Anthony Chen retweeted
Super excited about this work, led by @YichuanM and @andylizf ! It is possible to completely remove HTML parsing by directly retrieving and reading web screenshots through VLM.
HTML parsing is a hidden bottleneck that causes significant complexity and information loss that nobody really pays attention to, and it's so exciting that VLM progress made it possible to remove it.
Please check out this demo as well which is really cool!!! pixelrag.ai
The web was never meant to be flattened into text.
Yet most web RAG systems start by parsing HTML --- a complex and lossy process.
🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels.
Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly.
PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%.
Why?
(1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals.
(2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever.
(3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines.
Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG.
Demo below 👇
Code: github.com/StarTrail-org/Pix…
Paper: github.com/StarTrail-org/Pix…
Playground: pixelrag.ai/
Anthony Chen retweeted
New paper out today, proving a novel theorem in algebraic geometry with an internal math-specialized version of Gemini. This was a collaboration between @GoogleDeepMind (Professor Freddie Manners and @GSalafatinos, hosted by the Blueshift team) and Professors Jim Bryan, Balazs Elek, and Ravi Vakil.
arxiv.org/abs/2601.07222
Anthony Chen retweeted
This is Gemini 3: our most intelligent model that helps you learn, build and plan anything.
It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵
Anthony Chen retweeted
Instructions/reasoning are now everywhere in retrieval - we want embeddings to do it all! 🚀
But... is it even possible? 🤔
Turns out, it's not possible for single-vector models 😱 theoretically and empirically! To make it obvious we OSS a simple eval SoTA models flop on!
🧵
Anthony Chen retweeted
What if you could not only watch a generated video, but explore it too? 🌐
Genie 3 is our groundbreaking world model that creates interactive, playable environments from a single text prompt.
From photorealistic landscapes to fantasy realms, the possibilities are endless. 🧵
Anthony Chen retweeted
BREAKING: Gemini 2.5 Pro is now #1 on the Arena leaderboard - the largest score jump ever (+40 pts vs Grok-3/GPT-4.5)! 🏆
Tested under codename "nebula"🌌, Gemini 2.5 Pro ranked #1🥇 across ALL categories and UNIQUELY #1 in Math, Creative Writing, Instruction Following, Longer Query, and Multi-Turn!
Massive congrats to @GoogleDeepMind for this incredible Arena milestone! 🙌
More highlights in thread👇
Think you know Gemini? 🤔 Think again.
Meet Gemini 2.5: our most intelligent model 💡 The first release is Pro Experimental, which is state-of-the-art across many benchmarks - meaning it can handle complex problems and give more accurate responses.
Try it now → goo.gle/4c2HKjf
Anthony Chen retweeted
Think you know Gemini? 🤔 Think again.
Meet Gemini 2.5: our most intelligent model 💡 The first release is Pro Experimental, which is state-of-the-art across many benchmarks - meaning it can handle complex problems and give more accurate responses.
Try it now → goo.gle/4c2HKjf
Historic.
In partnership with Google, CHM is excited to announce the public release and long-term preservation of the source code for AlexNet, the neural network that revolutionized AI.
Learn more: hubs.li/Q03cL8-40
CHM’s GitHub access to open-source code: hubs.li/Q03cL6Dp0
GDM's work converting Gemini into a SOTA dual encoder is now out! SOTA results across all benchmarks including exceptionally strong coding performance.
Check out the tech report for more details and some interesting ablations!
🎉 Gemini Embedding is LIVE! 🎉
Try our state-of-the-art text embedding model for FREE on Vertex AI (text-embedding-large-exp-03-07; 120 QPM) & AI Studio (gemini-embedding-exp-03-07)!
➡️ APIs: bit.ly/gem-embed-vertex, bit.ly/gem-embed-aistudio
➡️ Report: bit.ly/gem-embed-paper
Anthony Chen retweeted
This is the most gut-wrenching blog I've read, because it's so real and so close to heart. The author is no longer with us. I'm in tears. AI is not supposed to be 200B weights of stress and pain. It used to be a place of coffee-infused eureka moments, of exciting late-night arxiv safaris, of wicked smart ideas that put smile on our faces. But all the incoming capital and attention seem to be forcing everyone to race to the bottom.
Jensen always tells us not to use phrases like "beat this, crush that". I absolutely love this perspective. We are here to lift up an entire ecosystem, not to send anyone to oblivion. I like to think of my work as expanding the pie. We need to bake the pie first, together, the bigger the better, before dividing it. It gives me comfort knowing that our team's works moved the needle for robotics, even just by a tiny bit.
AI is not a zero sum game. In fact, it is perhaps the most positive-sum game that humanity ever plays. And we as a community should act this way. Take care of each other. Send love to "competitors" - because in the grand schemes of things, we are all coauthors of an accelerated future.
I never had the privilege to know Felix irl, but I loved his research taste and set up Google Scholar alert for every one of his new papers. His works in agents and VLMs had a big influence on mine. He would've been a great friend. I want to get to know him, but I couldn't any more.
RIP Felix. May the next world have no wars to fight.
Thanks for sharing our paper!
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Google DeepMind conducts a deep performance analysis of long-context LLMs on in-context retrieval and reasoning.
They first present a benchmark with real-world tasks requiring 1M token context.
Report that long context LLMs can rival state-of-the-art retrieval and RAG systems, without any explicit training on the tasks.
Suggests that compositional reasoning (required in SQL-like tasks) is still challenging for these LLMs.
They also encourage the need for continued research on advanced prompting strategies as they noted significant boosts in performance when applying them for long context problems.
Just imagine what will be possible when we can get to 10 million context windows. The promise of using long-context LLMs without relying on external tools like retrieval systems or databases is of huge interest but has a long way to go.
Anthony Chen retweeted
Thrilled to unveil LOFT, our latest research showing how long-context language models like Gemini can subsume retrieval, RAG and more, right out of the box. This is the dream-retriever I've always wanted! No special training, amazing reasoning, few-shot promptable...🤯
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
Anthony Chen retweeted
Very happy to contribute to the Multimodal benchmarking on the LOFT project! Very excited to see with fewshot prompting only, Gemini 1.5 pro can already outperform CLIP on Coco, Flickr, by a large margin on 1M context tokens!
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
Thanks for sharing our work!
Google presents Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Long-context LM:
- Often rivals SotA retrieval and RAG systems
- But still struggles with areas like compositional reasoning
repo: github.com/google-deepmind/l…
abs: arxiv.org/abs/2406.13121
Anthony Chen retweeted
"just put the corpus into the context"!
Long context models can already match or beat various bespoke pipelines and infra in accuracy on non-trivial tasks! Hadn't expected this so soon, and honestly was hoping to milk RAG impact for a little longer 🤪
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
Anthony Chen retweeted
Do long-context LMs obsolete retrieval, RAG, SQL and more? Excited to share our answer! arxiv.org/abs/2406.13121 from the team at @GoogleDeepMind that wrote one of the 1st papers on RAG (REALM) and repeat SOTA on retrieval (Promptagator, Gecko). w/ Gemini 1.5 Pro, the answer is 🧵
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
Anthony Chen retweeted
Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?
Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!
arxiv.org/abs/2406.13121
That’s it for twitter! Check out our paper “Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?” and download the data in LOFT to see how LCLMs perform for yourself!
Paper: arxiv.org/abs/2406.13121
Data: github.com/google-deepmind/l… [9/10]
Work done with a great group of collaborators:
@leejnhk @zhuyundai @ddua17 @devendr06654102 @michaelboratko @yiluan9 @seba1511 @vincentperot @siddalmia05 @hexiang_hu @xudong_lin_ai @icepasupat @amini_aida @jeremy_r_cole @riedelcastro @iftekharnaim @mchang21 @kelvin_guu [10/10]