Your open-source AI ally. We are committed to making production-grade AI inference as accessible and reliable as electricity, powered by vLLM.
Joined October 2023
- Tweets616
- Following1.4K
- Followers1.2K
- Likes978
EmbeddedLLM retweeted
🎉 Step 5 Preview has Day-0 support in vLLM.
Sparse attention, MoE, multi-token prediction and vision, with serving work across Blackwell and Hopper.
The work spans the sparse indexer, FP8 MegaMoE and cache transfer. Here is how the pieces fit together, with kernel results and validation below. 🧵
1/6
EmbeddedLLM retweeted
🎉 Day-0 support for Ling-3.1-flash is here in vLLM!
~560B total parameters, ~25B active per token, and up to 1M context. From a rough brief to websites, mobile UIs, and interactive prototypes—serve it on your own infrastructure with vLLM.
Congrats to @AntLingAGI!
EmbeddedLLM retweeted
ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet.
MegaKernels is a technique that is massively important for high-interactivity decoding by reducing the # of kernel launches by 10x.
This marks an❗️important milestone❗️for ROCm vLLM to land major ROCm-exclusive features. CUDA vLLM doesn't even have draft PRs for upstream vLLM megakernel support yet.
EmbeddedLLM retweeted
It was an incredible honor to receive the National Medal of Science from @POTUS today.
My parents came to America with big dreams and a deep belief in education and hard work. This country gave me the opportunity to pursue my passion for semiconductors and work with extraordinary people to push the boundaries of what is possible.
I am so excited about what we can accomplish together as America continues to lead the world in science and technology.
.@POTUS presents the National Medal of Science to Sergey Brin, @JensenHuang, @elonmusk, and @LisaSu, and the National Medal of Technology and Innovation to @MichaelDell and @satyanadella.
EmbeddedLLM retweeted
AMD MI355X posts the best margin on every model in SemiAnalysis InferenceX agentic profit estimates:
Kimi K3: 45.7% vs 43.8% (GB300)
GLM 5.3: 63.4% vs 63.2% (GB200)
MiniMax M3: 51.5% vs 48.4% (B300)
DeepSeek V4 Pro: 87.9% (no green data)
inferencex.semianalysis.com
EmbeddedLLM retweeted
Zero mismatches. Bit-for-bit aligned. 🔥
Incredibly proud of our team's work integrating with vime @vllm_project. Rock-solid RL pipelines are now a reality on @AMD ROCm.
Read the deep dive below and star our repo⭐ github.com/RL-Align/RL-Kerne… – at Singapore
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Massive shoutout to TJ & Team at @EmbeddedLLM; they are the lead maintainers & CODEOWNERS of vLLM ROCm! Up until May 2026, they did not even have persistent access to an MI355X cluster, but after SemiAnalysis worked with & convinced AMD leadership of something that is quite obvious in hindsight, they finally got a persistent cluster to make MI355X great on AMD!
Super proud of the @EmbeddedLLM Team, highly respected within the @vllm_project community.
EmbeddedLLM retweeted
🎉 Congrats to @XiaomiMiMo on MiMo-V2.6: day-0 support in vLLM for both sizes.
Two MoEs with text, image, video and audio in one checkpoint: Pro is 1.02T total with 42B active, Flash is 309B with 15B active. 1M context, native FP8 weights. Built-in DFlash spec decoding drafts 7 tokens a step.
Same architecture class as V2.5, and the serving pieces carry over: hybrid sliding-window plus global attention, the mimo reasoning and tool-call parsers, DFlash spec decoding.
Thanks to the @XiaomiMiMo team for opening the weights and the RL stack! 🙌
🔗 recipes.vllm.ai/XiaomiMiMo/M…
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：mimo.xiaomi.com/mimo-v2-6
EmbeddedLLM retweeted
10 things to know before joining the AMD AI Developer Program 👇
1. It’s free. Anyone can join, whether you are a student starting out or a seasoned ML engineer.
2. You get $100 in cloud credits on day one. Enough to prototype, benchmark, and get moving on AMD tech.
3. You don’t need any hardware to start. Build immediately with cloud access.
4. There’s a private Discord with a community of engineers who will provide technical discussion and help.
5. You get early invites to dev events and workshops. Before they’re public. Before they fill up.
6. Monthly Ask the Expert hours with open-source partners. Ask questions. Go deep. Learn the tricks of the trade.
7. Your projects can get featured on official AMD Developer channels and at events.
8. You’re automatically entered into monthly hardware raffles. No forms. No extra steps.
9. You get a free month of @DeepLearningAI Pro. High quality courses for you to expand your skills.
10. The more you participate, the more you unlock. More access, more visibility, more opportunities.
If you’re building, training, or optimizing AI models, we’d love to have you join us!
EmbeddedLLM retweeted
🔎 New blog: day-0 support gets a model running in vLLM. From there, the community keeps making it faster.
@AMD and @EmbeddedLLM wrote up that process for MiniMax M3 on Instinct MI355X: follow the bottleneck.
Read it for the 3–4× serving gains and performance tips you can reuse on the next model optimization. 👇
🔗 vllm.ai/blog/2026-09-10-mini…
EmbeddedLLM retweeted
Model Runner V2 is the default runner for all models as of vLLM v0.29.0. 🎉
594 commits from 277 contributors, 91 of them first-timers.
Highlights:
🌙 Another optimization round on Kimi-K3, latent tail down to Mamba metadata
🐳 DeepSeek-V4 shared experts fused into MegaMoE
🧠 Mamba prefix caching keeps internal prefill checkpoints: 9~25% better TTFT
⚡ Speculative decoding reports per-request acceptance stats over the API
🔄 RL weight sync adds the sharded_rdt P2P backend: each worker pulls only its own TP/EP slice
🆕 New models: Hy4-preview, Qwen3.8-Flash-Next, GraniteSWA, NemotronH Omni Reasoning V3
Thread 👇
EmbeddedLLM retweeted
New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving.
Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, @SemiAnalysis_'s public agentic benchmark.
🧵1/6
EmbeddedLLM retweeted
ALERT🚨: AMD INCREASED @vllm_project PERFORMANCE BY 11x IN LESS THAN 19 DAYS ON MI355X AGENTIC WORKLOADS on the modern MiniMax M3 model!
This was done entirely through software optimizations, mainly by optimizing the long-context attention op, along with other optimizations! This is the power of the ROCm stack: buy the hardware once, get performance updates over time through software optimizations for free.
EmbeddedLLM retweeted
On total tokens per $ TCO, a new AMD MI355x submission beats B300 at lower interactivity ranges on AgentX. Shoutout to vLLM, AMD, and LMCache engineers.
EmbeddedLLM retweeted
Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵
EmbeddedLLM retweeted
K2-Horizon has day-0 support in vLLM, and IFM released intermediate checkpoints, detailed data-construction recipes, the training code, and fine-grained logs alongside the weights. 📷
512K context and Apache-2.0 from 3.7B up, with reasoning and tool calling in the checkpoints for agents and long documents.
vLLM runs the whole family, from local to large-scale data centers. Prototype on the small end, scale up without changing the stack.
Thanks @IFM_AI for opening all six and every stage that led to them, not just the last checkpoint! 📷
📷 recipes.vllm.ai/IFM
Introducing K2 Horizon: a connected fleet of six foundation models ranging from 0.9 billion to 375 billion parameters.
- Frontier performance: Across coding and agentic tasks, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B and 7B models setting new state of the art at their respective scales.
- Radical openness: K2 Horizon represents the largest fully open-source model launch in AI history. The fully open code, training data and recipes are a significant step forward in transparency.
Launch page: ifm.ai/k2/
Tech blog: ifm.ai/blog/k2
Hugging Face: huggingface.co/collections/I…
EmbeddedLLM retweeted
🎉 Congrats to @deepseek_ai on DeepSeek-V4-Flash-Vision-Exp, the first multimodal model in the V4 family! vLLM serves it now.
A vision encoder and aligner on the 285B/13B V4-Flash MoE backbone, with text-agent scores holding steady. 🚀
🔗 recipes.vllm.ai/deepseek-ai/…
DeepSeek-V4-Flash-Vision-Exp is now live on the DeepSeek API Platform! 🚀
🔹 This experimental multimodal model matches DeepSeek-V4-Flash on text capabilities—including agents, reasoning, and world knowledge.
🔹 On multimodal agent benchmarks, V4-Flash-Vision-Exp makes a major leap over V4-Flash, bringing multimodal agent performance close to Opus-4.8.
Try it with model='deepseek-v4-flash-vision-exp'. DeepSeek Harness 0.1.1 was released today with out-of-the-box support for the new model.
1/n
EmbeddedLLM retweeted
🔐 AI security has a new boundary: data in use.
As models remember more through KV cache, security needs to protect not just data at rest or in transit, but the memory powering inference.
Explore how we're helping secure stateful AI inference. 👇
weka.ly/4cMS7ZZ
EmbeddedLLM retweeted
🔥 SPEAKER ANNOUNCEMENT | @vllm_project Bangkok Day
(Thailand's largest event dedicated to vLLM)
It is a real honor to announce this one.
As the organizer, I'm proud to introduce one of the speakers I'm most excited about:
Tun Jian Tan
Core Contributor of vLLM
Principal AI Engineer @EmbeddedLLM
And here's what makes it special: Tun Jian is flying to Bangkok to join us onsite, not virtually.
🎙️ Session: Introduction to vLLM: High-Performance LLM Serving & Maximizing Performance on Limited GPUs
This one comes from someone who writes the code. It starts with vLLM's core architecture and how the engine actually works, then goes straight at the question most teams here live with: how do you get more out of the GPUs you already have?
What you'll take home:
- vLLM's core architecture and the features behind fast, efficient serving
- Practical techniques you can apply on Monday: DFlash, KV-cache compression, KV offloading, prefix caching, chunked prefill, sleep mode and smarter batching
- More importantly, when to reach for each one and what it costs you, instead of turning on every flag and hoping for the best
Why this session matters for Thailand:
GPU access is the constraint here. Most teams are not choosing between fleets, they are trying to make the cards they have go further. This session answers exactly that, from a core contributor of vLLM. Whether you're new to vLLM or already running production workloads, the goal is simple: understand vLLM and get more performance from every GPU you have.
📅 Sunday, 6 September 2026, 2:00 PM to 7:00 PM
📍 JAS Green Village Ramkhamhaeng, 3rd floor (JAI Arena), Bangkok
🎟️ Free to attend. All sessions delivered in English.
Huge thanks to our partners Inferact, Embedded LLM, Red Hat, Tensormesh, and AGICAFET CO., LTD., for making the largest AI Inference event in Thailand happen.
"One vLLM Always vLLM"
#vLLMBangkok #vLLM #EmbeddedLLM #AIInfrastructure #LLMServing #KVCache #Inference #AGICAFET
