Research Scientist @ Google. Past: PostDoc at UoE. PhD in NLP/MT @edinburghnlp. All opinions are my own.
Joined March 2018
- Tweets221
- Following331
- Followers734
- Likes806
Biao Zhang retweeted
Beam is now available across platforms, with open weights under the Apache 2.0 license - in collaboration with an ecosystem of 40+ partners.
We're releasing:
- Weights, tokenizer, and reference implementation
- BF16, FP8 and NVFP4 weights and a DFlash2 drafter
- The open source harness we built and trained Beam on: Mirror CLI
Weights and model card on Hugging Face: huggingface.co/reflection
Yes, we used depth scaled residual
reflection beam oss architecture details
IT'S SLIDING WINDOW AGAIN, same window size and pattern as inkling (window 512 and 5:1, meta is 2k and 3:1). they use a similar global attention scaling as llama4 (arxiv.org/abs/2501.19399) which still makes it position dependent even if they don't use RoPE
they use a residual depth scaling, quite similar to muse open but "fixed", tho it's not really fixed because you have a learned norm nearby. you can actually plot both the effective residual contributions and you see that it's not super different from muse
they use gated attention (with a per head bias for the full attention layer), QK Norm, Sandwich norm, similar softcapping before the LM head (like gemma and also muse), no shared experts
the shape is pretty similar to other american oss open weight models (but tbf, mimo v2 flash and stepfun 3.5 and 5 also have this shape, it's not only us labs 👀)
model weight: huggingface.co/reflection/Be…
Biao Zhang retweeted
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: reflection.ai/beam
Biao Zhang retweeted
I am excited to share some of the progress we are making towards using Gemini to accelerate scientific discovery in the real-world.
We present an extension of Co-Scientist that transitions from pure in silico ideation to execution-grounded research partner, adapting to multiple levels of autonomy across different scientific domains.
Biao Zhang retweeted
Introducing T5Gemma 2, the next generation of encoder-decoder models 🚀
Built on top of Gemma 3, we were able to build compact models at sizes of 270m-270m, 1B-1B, and 4B-4B sizes.
While most models today are decoder-only, T5Gemma 2 is the first (I'm aware of) multimodal, long-context, and heavily multilingual (140 languages) encoder-decoder model out there.
We hope this model enables the model research community as well as the community of devs ready to explore with new architectures.
Blog: blog.google/technology/devel…
Models: huggingface.co/collections/g…
Paper: arxiv.org/abs/2512.14856
Biao Zhang retweeted
We just release 2 new open-weight Gemma models. FunctionGemma and T5Gemma optimized for on-device agentic actions and multimodal applications.
FunctionGemma
🤖 270M parameter built for on-device tool use.
📂 32K token context window.
📱 85% accuracy on mobile system call identification.
🧠 Trained on 6 trillion tokens.
T5Gemma 2
🖼️ Multimodal encoder-decoder architecture handling both text and image inputs.
🌐 128K context window across over 140 languages.
📏 Available in three sizes: 270M, 1B, and 4B parameters.
👁️ Normalizes images to 896x896 resolution, encoded into 256 tokens each.
Biao Zhang retweeted
Meet T5Gemma 2, the next evolution of Google's encoder-decoder family! 🚀
Building on Gemma 3, these models bring major upgrades to efficiency and capability:
🖼️ Multimodal: Understands images + text out of the box.
📚 128K Context: Handles massive datasets with long-context support.
🌍 140+ Languages: Massive multilingual training.
⚡ Efficient Architecture: New tied embeddings & merged attention for faster inference and smaller footprints (270M, 1B, and 4B sizes).
Check out the pre-trained checkpoints on Kaggle and Hugging Face now! 🛠️✨
Today, we are thrilled to release T5Gemma 2, the next generation of T5Gemma with multilingual, multi-modal, and long-context capabilties.
Read the announcement👉 blog.google/technology/devel…
Introducing T5Gemma 2, the next generation of encoder-decoder models 🚀
Built on top of Gemma 3, we were able to build compact models at sizes of 270m-270m, 1B-1B, and 4B-4B sizes.
While most models today are decoder-only, T5Gemma 2 is the first (I'm aware of) multimodal, long-context, and heavily multilingual (140 languages) encoder-decoder model out there.
We hope this model enables the model research community as well as the community of devs ready to explore with new architectures.
Blog: blog.google/technology/devel…
Models: huggingface.co/collections/g…
Paper: arxiv.org/abs/2512.14856
That foundation inspired T5Gemma, our first recipe for adapting modern strong decoder-only models into encoder-decoder models🔗 arxiv.org/abs/2504.06225
Now we extend it to the multimodal and long-context regime with T5Gemma 2!
Biao Zhang retweeted
Introducing T5Gemma 2, the next generation of encoder-decoder models, built on the powerful capabilities of Gemma 3.
Key innovations and upgraded capabilities include:
+ Multimodality
+ Extended long context
+ Support of 140+ languages out of the box
+ Architectural improvements for efficiency
+ And more
blog.google/technology/devel…
Biao Zhang retweeted
Introducing T5Gemma 2, the next generation of encoder-decoder models 🚀
Built on top of Gemma 3, we were able to build compact models at sizes of 270m-270m, 1B-1B, and 4B-4B sizes.
While most models today are decoder-only, T5Gemma 2 is the first (I'm aware of) multimodal, long-context, and heavily multilingual (140 languages) encoder-decoder model out there.
We hope this model enables the model research community as well as the community of devs ready to explore with new architectures.
Blog: blog.google/technology/devel…
Models: huggingface.co/collections/g…
Paper: arxiv.org/abs/2512.14856
Biao Zhang retweeted
Introducing EmbeddingGemma, our newest open model that can run completely on-device. It's the top model under 500M parameters on the MTEB benchmark and comparable to models nearly 2x its size – enabling state-of-the-art embeddings for search, retrieval + more.
Biao Zhang retweeted
Introducing EmbeddingGemma, our new open embedding model for on-device AI applications.
- Highest ranking open model under 500M on the MTEB benchmark.
- Runs on less than 200MB of RAM with quantization.
- Dynamic output dimensions from 768 down to 128.
- Input context length of 2048 tokens.
- Trained on over 100 languages.
- Based on Gemma 3 270M.
Start building today with @GoogleDeepMind Embedding Gemma and @huggingface Sentence Transformers, @ollama, Llama.cpp, MLX, @lmstudio, @weaviate_io, @googlecloud Vertex AI, @AMD, @baseten, @Cloudflare, @nvidia, and more.
Biao Zhang retweeted
Introducing EmbeddingGemma🎉
🔥With only 308M params, this is the top open model under 500M
🌏Trained on 100+ languages
🪆Flexible embeddings (768 to 128 dims) with Matryoshka
🤗Works with your favorite open tools
🤏Runs with as little as 200MB
developers.googleblog.com/en…
Biao Zhang retweeted
Introducing EmbeddingGemma: our new open, state-of-the-art embedding model designed for on-device AI 📱
From encoder-decoder to world-class embeddings! 🚀
Super excited to introduce EmbeddingGemma, our new open embedding model. It leverages the strong encoder from its sibling, T5Gemma, to achieve new SOTA performance for models under 500M!
Dive in 👇
Biao Zhang retweeted
Our Google Translate team is bringing a strong presence to #ACL2025 in Vienna this week! 🇦🇹 My group is excited to present several of our latest papers. 👇 Don't miss them!
Biao Zhang retweeted
Voxtral uses online DPO!
In our continued commitment to open-science, we are releasing the Voxtral Technical Report: arxiv.org/abs/2507.13264
The report covers details on pre-training, post-training, alignment and evaluations. We also present analysis on selecting the optimal model architecture, which pre-training format to use, and the benefits of DPO.
