Manage GPU clusters for running LLMs https://xitter.cf/t.co/UUXf6jWiBY

Joined June 2024
🚀 DeepSeek-V4.1-Flash-1017 is now supported on GPUStack! Get started with self-hosted DeepSeek on your own infrastructure, with full control over where your model runs. 🔒 👉 github.com/gpustack/gpustack
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
28
A quick translation and a payment-system design shouldn’t need the same model. GPUStack × Jev routes tasks across self-hosted Qwen and DeepSeek through one endpoint. Here’s the setup—and how we checked which model actually answered.
1
49
Bring your model mix and a task you’re figuring out how to route. Join the GPUStack Discord to compare setups and routing policies with the community. 👉discord.gg/VXYJzuaqwD
33
Keeping trajectory branching and training costs under control—definitely a direction worth watching~
Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training. ✨ It launches with two core designs: 🥳 - Elastic Scheduler: reallocates GPUs between rollout & training as demand shifts—live executions never interrupted. Up to 1.85× speedup over Colocate, 1.78× over Async. - Trajectory Processor: turns branching executions (compaction, sub-agents, retries) into prefix-shared trajectory trees with bounded, deduplicated training samples. Partial progress after timeout still gets scored. Scaled to Qwen3.8-2.4T-A95B with ~700K tokens per rollout, QwenGyre lifts NL2RepoBench from 52.5% to 58.5% in 48 steps, cutting end-to-end training time by ~44% vs Async. 🔗 Read the paper: arxiv.org/abs/2609.33848
68
One A800 80GB, from deployment to image generation and editing. 🍀Here’s our Qwen-Image-2.1 walkthrough with GPUStack and vLLM-Omni, starting with backend setup and model deployment. Preview: replacing coffee with books
2
146
With the model running, let’s generate an image. Send your prompt and generation settings to /v1/images/generations. 👇The example request and output are below.Use your service URL, API key, and deployed model name.
1
80
From generation to editing: pass the image back with “Replace the cup of coffee with a stack of books.” ☕➡️📚 Explore GPUStack and try it yourself: gpustack.ai github.com/gpustack/gpustack
1
20
Building a self-hosted AI inference stack? GPUStack unifies model deployment, GPU management, and API routing across engines like vLLM and SGLang—with monitoring and governance for day-to-day operations. Explore the architecture below. 👇
🤖 Made with AI
2
162
Curious how well cross-modal search works in practice. Would love to see more demos
Meet EmbeddingGemma 2, our first natively multimodal open model for on-device embeddings. It expands beyond text to unify code, images, audio, and video in a shared space. 🧵
1
136
🧐Looking for a self-hosted AI inference platform? GPUStack brings model deployment and heterogeneous GPU management into one place, with pluggable inference engines including vLLM and SGLang. 👉Explore the project: github.com/gpustack/gpustack
2
100
🚀 GPUStack v2.3 preview is here! Try v2.3.0rc2: vGPU (K8s only), automatic PD disaggregation, cross-replica KV cache sharing & more. 😍What are you most excited to try? Share your feedback! ⚠️ Use a fresh environment. Upgrades aren't supported. #GPUStack #AI #OpenSource
🤖 Made with AI
1
4
118
What's new? 👇 🧩 vGPU ⚡ Automatic PD disaggregation 🔄 KV cache sharing 📦 Model & backend OTA ⏰ Scheduled scaling 📊 Intelligent benchmarks 🔀 Advanced routing 🛠️ Deployment management We'd love your feedback!
56
🤩Multimodal workloads bring new serving challenges. At GPUStack, we’re exploring these in practice with Qwen-Image deployment using vLLM-Omni. Excited to see the team advancing a unified runtime for multimodal generation!
🚀 Excited to share the vLLM-Omni technical report: a unified serving runtime for omni-modality generation. 📄 Paper: lnkd.in/gM264Pdb 🔗 Repo: lnkd.in/gbXrTjhS Speech assistants, visual generation, world models, and robot loops have pushed serving past a single text decode loop. The execution patterns diverge with the output: multi-stage autoregressive pipelines, iterative diffusion, and sessions that carry state from step to step. LLM servers and diffusion stacks each go deep on only one of these, so deployments fall back to stitching disjoint runtimes together. vLLM-Omni is the shared control plane for that mix. An orchestrator advances each request across stages; specialized engines run the compute; a connector carries the payloads; and the same session path keeps duplex, world-model, and robot loops on one runtime. Really grateful to the vLLM and vLLM-Omni teams for the collaboration and support. Contributions are welcome. 🙏
3
96
Reliable AI agents need to recover when things go wrong. NVIDIA’s PivotOPD trains agents to both avoid pivotal mistakes and get back on track. At GPUStack, we’re following how advances in agent capabilities and inference infrastructure support increasingly complex AI workloads.
An AI agent makes a mistake early in a task, then keeps going in the wrong direction. Our researchers built PivotOPD to teach agents how to avoid those mistakes and recover when they happen. During training, a teacher model shows the agent a better action and how to get back on track over the next few steps. Read the paper and watch how it works: research.nvidia.com/labs/lpr…
62
Congrats on the launch! MiMo-V2.6 Pro and Flash bring an exciting combination of omnmodal capabilities and strong agent performance. Looking forward to seeing them deployed and served across diverse GPU environments with GPUStack. 🚀
Introducing Xiaomi MiMo-V2.6 — Pro & Flash. Frontier intelligence, all the modalities, built in public. 🔹 Two omnimodal models, advancing through scaled reinforcement learning 🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks 🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models 🔹 Stronger coding, computer use, 3D reasoning and creative capabilities 🔹 Open model weights, technical report, RL environments and training code Blog:mimo.xiaomi.com/mimo-v2-6
1
112
Deploying DeepSeek-V4.1-Flash on Day 0 🚀 GPUStack × vLLM benchmarks across H20, H200, and H100—covering deployment setup and performance, all in one place. 🔗 Get started with GPUStack: github.com/gpustack/gpustack #GPUStack #DeepSeek #vLLM #LLMDeployment #AIInfrastructure
1
3
193
GPUStack Day 0 support for Kimi-K3 @Kimi_Moonshot 🚀 We benchmarked vLLM vs. SGLang on a single 8×NVIDIA B300 node: • 64K: vLLM was 1.50× faster • 200K: SGLang was 1.31× faster • The crossover came from Decode scalability and DCP—not Prefill #GPUStack #KimiK3 #vLLM #SGLang
183
Congratulations to the Kimi team @Kimi_Moonshot on the open release of Kimi K3! 🎉 GPUStack is ready with Day 0 support—give it a try and start deploying Kimi K3 today. More open models, more possibilities. Let’s keep building in the open! 🚀
Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale. Model weights: huggingface.co/moonshotai/Ki… Tech report: github.com/MoonshotAI/Kimi-K… Tech blog: kimi.com/blog/kimi-k3
1
184
Congrats on the release! @thinkymachines Exciting to see Inkling pushing open multimodal models forward. We just ran Inkling-BF16 on GPUStack with 8× H20-141G — great to see it working in a real-world deployment. #Inkling #GPUStack #AIInfra #OpenSourceAI #MultimodalAI #LLM
Today, we are introducing Inkling. Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available. thinkingmachines.ai/news/int… Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
2
219
Where did your GPU resources go? With the new GPUStack Usage feature, you can now track: ✅ Token usage ✅ GPU/CPU instance runtime ✅ Storage capacity usage ✅ Usage by user, model, and API key ✅ Resource lifecycle events Read the full blog 👇 [dev.to/gpustack/where-did-yo…]
1
150