Your open-source AI ally. We are committed to making production-grade AI inference as accessible and reliable as electricity, powered by vLLM.
Joined October 2023
- Tweets616
- Following1.4K
- Followers1.2K
- Likes978
Pinned Tweet
Tomorrow at #SFTechWeek with @tensorwave, @CosmicLabsTech, @weka and @makora. Hardware, watts, kernel and token.
Silicon doesn't serve itself. That's the layer we run.
10am, SF. If you're shipping tokens, meet our team.
Our @Techweek_ Inference at SCALE panel is growing!!
Excited to have our friends from @makora, @weka, and @EmbeddedLLM joining our @tensorwave x @CosmicLabsTech teams for a conversation around hardware, inference, market dynamics and more.
partiful.com/e/Xb2BSFjKC6Wtw…
We helped close the performance gap.
@SemiAnalysis_ Kimi K3 estimator puts MI355X at 36% higher revenue per chip-hour than HGX B300, with a 38.8% profit margin vs 7.7%.
Performance = profit.
🚀 @AMD MI355X just showed competitive performance to NVIDIA B300 on @Kimi_Moonshot K3 serving!
8× MI355X delivers 24.4% more throughput than HGX B300 (7,567 vs 6,083 tok/s/GPU) at 50 tok/s/user on @SemiAnalysis_ AgentX agentic coding benchmark, and 87.4% more tokens per dollar.
We cover the vLLM optimizations, AMD kernels, and hardware details that closed the gap, plus patterns for spotting bottlenecks so you can get more out of vLLM on your own GPUs.
Thanks @AMD @embeddedLLM @SemiAnalysis_
Full writeup: 🔗 vllm.ai/blog/2026-10-16-kimi…
EmbeddedLLM retweeted
🎉 Step 5 Preview has Day-0 support in vLLM.
Sparse attention, MoE, multi-token prediction and vision, with serving work across Blackwell and Hopper.
The work spans the sparse indexer, FP8 MegaMoE and cache transfer. Here is how the pieces fit together, with kernel results and validation below. 🧵
1/6
EmbeddedLLM retweeted
🎉 Day-0 support for Ling-3.1-flash is here in vLLM!
~560B total parameters, ~25B active per token, and up to 1M context. From a rough brief to websites, mobile UIs, and interactive prototypes—serve it on your own infrastructure with vLLM.
Congrats to @AntLingAGI!
Building the future of AI on @AMD takes the whole stack working in lockstep.
From @tensorwave's Instinct GPU clusters and @weka's data layer to our work at @EmbeddedLLM pushing vLLM on ROCm and TokenVisor into production, the shift from raw GPU hours to governed, high-throughput inference and agent services is happening right now.
Great sharing the stage with @CosmicLabsTech, @makora_ai, @miguelsaavedra, and Prithvi Mattur during #SFTechWeek! 🚀
Thank you to everyone who came out for our “From Training to Inference” panel with @CosmicLabsTech, @makora_ai, @EmbeddedLLM, and @weka during #SFTechWeek.
It was a packed room and a great conversation about where AI compute is heading, what builders need now, and how the infrastructure behind it all is evolving.
Shoutout to our speakers @miguelsaavedra and Prithvi Mattur!
ALT “From Training to Inference” panel with Cosmic Labs, Makora, Embedded LLM, and Weka.
EmbeddedLLM retweeted
ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet.
MegaKernels is a technique that is massively important for high-interactivity decoding by reducing the # of kernel launches by 10x.
This marks an❗️important milestone❗️for ROCm vLLM to land major ROCm-exclusive features. CUDA vLLM doesn't even have draft PRs for upstream vLLM megakernel support yet.
EmbeddedLLM retweeted
It was an incredible honor to receive the National Medal of Science from @POTUS today.
My parents came to America with big dreams and a deep belief in education and hard work. This country gave me the opportunity to pursue my passion for semiconductors and work with extraordinary people to push the boundaries of what is possible.
I am so excited about what we can accomplish together as America continues to lead the world in science and technology.
.@POTUS presents the National Medal of Science to Sergey Brin, @JensenHuang, @elonmusk, and @LisaSu, and the National Medal of Technology and Innovation to @MichaelDell and @satyanadella.
Meet us at #COLM2026 in San Francisco.
Oct 9 at DAIH: our research on deploying clinical AI where the data lives across Nvidia and AMD GPU, with energy and cost in view.
Building models or putting them into production? Come say Hi! @manpree59181175
DM us to meet.
EmbeddedLLM retweeted
AMD MI355X posts the best margin on every model in SemiAnalysis InferenceX agentic profit estimates:
Kimi K3: 45.7% vs 43.8% (GB300)
GLM 5.3: 63.4% vs 63.2% (GB200)
MiniMax M3: 51.5% vs 48.4% (B300)
DeepSeek V4 Pro: 87.9% (no green data)
inferencex.semianalysis.com
Massive respect to @AMD, @roaner and @AnushElangovan for investing where it counts: dedicated hardware, CI, and the room for core maintainers @Rxday000 to push ROCm to its limit.
The partnership is in a better place than it has ever been. 502 commits is the baseline. When software and silicon move at the same speed, the whole open source ecosystem wins. Proud of what we’re building together on @vllm_project.
#1 company contributing code to @vllm_project right now?
@AMD: 502 commits to core vLLM in 90 days. 13% of all org contributions, almost 4x Nvidia.
Also grateful to Red Hat, IBM, Embedded LLM & Inferact for building it with us.
Open source wins when hardware has a choice.
EmbeddedLLM retweeted
Zero mismatches. Bit-for-bit aligned. 🔥
Incredibly proud of our team's work integrating with vime @vllm_project. Rock-solid RL pipelines are now a reality on @AMD ROCm.
Read the deep dive below and star our repo⭐ github.com/RL-Align/RL-Kerne… – at Singapore
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Massive shoutout to TJ & Team at @EmbeddedLLM; they are the lead maintainers & CODEOWNERS of vLLM ROCm! Up until May 2026, they did not even have persistent access to an MI355X cluster, but after SemiAnalysis worked with & convinced AMD leadership of something that is quite obvious in hindsight, they finally got a persistent cluster to make MI355X great on AMD!
Super proud of the @EmbeddedLLM Team, highly respected within the @vllm_project community.
EmbeddedLLM retweeted
🎉 Congrats to @XiaomiMiMo on MiMo-V2.6: day-0 support in vLLM for both sizes.
Two MoEs with text, image, video and audio in one checkpoint: Pro is 1.02T total with 42B active, Flash is 309B with 15B active. 1M context, native FP8 weights. Built-in DFlash spec decoding drafts 7 tokens a step.
Same architecture class as V2.5, and the serving pieces carry over: hybrid sliding-window plus global attention, the mimo reasoning and tool-call parsers, DFlash spec decoding.
Thanks to the @XiaomiMiMo team for opening the weights and the RL stack! 🙌
🔗 recipes.vllm.ai/XiaomiMiMo/M…
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：mimo.xiaomi.com/mimo-v2-6
The best kept secret on the industry. Profit margin exceeds GB200 NVL72 and B300.
Proud of our team and our partners at @AIatAMD!
AMD ALERT🚨🚨: On Kimi K3 2.8T, AMD has even better profit margins per gigawatt compared to B300, GB200 NVL72, and B200. This is due to AMD's lower TCO and better performance, leading to better profit margins of 53.6% on Mi355X compared to 44.3% on GB200 NVL72. This is from our open-source Agentic Inference benchmark, AgentX.
Shoutout to @roaner & @EmadBarsoumPi 10x engineers for optimizing this 2.8T model!
EmbeddedLLM retweeted
10 things to know before joining the AMD AI Developer Program 👇
1. It’s free. Anyone can join, whether you are a student starting out or a seasoned ML engineer.
2. You get $100 in cloud credits on day one. Enough to prototype, benchmark, and get moving on AMD tech.
3. You don’t need any hardware to start. Build immediately with cloud access.
4. There’s a private Discord with a community of engineers who will provide technical discussion and help.
5. You get early invites to dev events and workshops. Before they’re public. Before they fill up.
6. Monthly Ask the Expert hours with open-source partners. Ask questions. Go deep. Learn the tricks of the trade.
7. Your projects can get featured on official AMD Developer channels and at events.
8. You’re automatically entered into monthly hardware raffles. No forms. No extra steps.
9. You get a free month of @DeepLearningAI Pro. High quality courses for you to expand your skills.
10. The more you participate, the more you unlock. More access, more visibility, more opportunities.
If you’re building, training, or optimizing AI models, we’d love to have you join us!
Tomorrow 14:00 GMT+8: TAIONE Fellowship briefing.
Paid track to contribute upstream to vLLM and other AI infra projects.
Mentored by @Rxday000. Students first.
Join: meet.mplusapp.com/m/joinMeet…
Tracks: taione.org/fellowship/tracks
We’re the technical partner on the vLLM track.
很多朋友在搞 AI Infra
上週去聽台大廖教授分享的，技術人不用跟別人卷啥 AI 應用
光搞 AI Infra 就夠吃十年了，還有開玩笑說，如果你十年內沒賺到錢，那就真的沒辦法了哈哈，可能十年後就真的很難，教授也說不定自己也會被取代
台灣的話最近台灣開源基金會剛成立，搞了個計劃「TAIONE Fellowship Program」
就專搞 Apache 開源的 AI Infra，讓大家一起討論、學習以及貢獻
最牛逼的是，成為 Fellow 後，每月可以領三萬台幣 👀 沒想到搞開源還能賺錢 超讚👍
taione.org/gosf/about
EmbeddedLLM retweeted
🔎 New blog: day-0 support gets a model running in vLLM. From there, the community keeps making it faster.
@AMD and @EmbeddedLLM wrote up that process for MiniMax M3 on Instinct MI355X: follow the bottleneck.
Read it for the 3–4× serving gains and performance tips you can reuse on the next model optimization. 👇
🔗 vllm.ai/blog/2026-09-10-mini…
EmbeddedLLM retweeted
Model Runner V2 is the default runner for all models as of vLLM v0.29.0. 🎉
594 commits from 277 contributors, 91 of them first-timers.
Highlights:
🌙 Another optimization round on Kimi-K3, latent tail down to Mamba metadata
🐳 DeepSeek-V4 shared experts fused into MegaMoE
🧠 Mamba prefix caching keeps internal prefill checkpoints: 9~25% better TTFT
⚡ Speculative decoding reports per-request acceptance stats over the API
🔄 RL weight sync adds the sharded_rdt P2P backend: each worker pulls only its own TP/EP slice
🆕 New models: Hy4-preview, Qwen3.8-Flash-Next, GraniteSWA, NemotronH Omni Reasoning V3
Thread 👇
EmbeddedLLM retweeted
New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving.
Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, @SemiAnalysis_'s public agentic benchmark.
🧵1/6