Ivan Rocha retweeted
Maya v1.1.69 is out: Hy4-Flash gets faster on every kind of PC. ⚡
✅ Drafting on 8-12 GB cards with a small DFly2 drafter (up to +29% on an RTX 3050 8 GB)
✅ AMD / Strix & Gorgon Halo: prompts 2-2.5x faster, drafting ~30% faster
✅ Long prompts on NVIDIA: up to +56% at 128K
✅ Contexts up to 1M on ordinary cards, with the cache in RAM
✅ Pictures at full size, even on 8 GB cards
🧪 Hy4-Flash on Windows + AMD, experimental
Drafting still checks every token. Thanks to @needmorevram, @superalesha and every contributor.
github.com/mw00/project-maya…
Ivan Rocha retweeted
Big update for GLM 5.3 Flash for 2x DGX Sparks
Running latest TensorFold (v1.3.0). Lots of changes:
- Faster decode: CUDA kernels now chain directly with fewer launch gaps (PDL)
- Cleaner stack: Our paged FP8 KV, MoE, and DFlash2 speedups are now built into upstream (100+ patches down to 13)
- Smarter prompt cache: Cache space is reserved up front, and multi-turn thinking chats reuse cached reasoning instead of re-reading history
- More stable 2-GPU vision: Long image prompts no longer time out the second GPU
Even more changes and improvements coming soon!
Get it here:
github.com/MiaAI-Lab/GLM-5.3…
Ivan Rocha retweeted
Tencent's 220B Hy4-Flash now runs on a home PC with an 8 GB graphics card. 🚀
Maya v1.1.68 is out, with four quants we made ourselves for @TencentHunyuan's new model. Each one is measured against the full-precision original:
Maya-XS · 45 GB · 88.5% same top token
Maya-S · 53 GB · 91.8%
Maya-M · 68 GB · 93.6%
Maya-L · 90 GB · 96.6%
How does 220B fit? What doesn't fit the graphics card runs on the CPU from RAM, what doesn't fit RAM streams from the SSD, and the model's 63 GB n-gram table stays on the SSD.
✅ Drafting: the model checks every drafted token
✅ Vision, thinking on or off, tool calls
✅ NVIDIA RTX 20 series and newer on Linux and Windows
✅ AMD on Linux, Strix Halo and Gorgon Halo included
✅ One setup command picks the size and context that fit your PC
Which one will you pick?
Full notes 👇
github.com/mw00/project-maya…
🚨🔥 𝗧𝗘𝗡𝗖𝗘𝗡𝗧 𝗛𝗬𝟰-𝗙𝗟𝗔𝗦𝗛 𝗦𝗔𝗚𝗘-𝗥𝗘𝗫𝗟𝟯 𝟰.𝟴𝟰 𝗕𝗣𝗪, 𝟵𝟲.𝟵𝟮% 𝗧𝗢𝗣-𝟭 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧, 𝗔𝗡𝗗 𝗦𝗜𝗭𝗘𝗗 𝗙𝗢𝗥 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🤯🚀
The fidelity results are looking GREAT!
🏆 𝟵𝟲.𝟵𝟮% 𝗧𝗢𝗣-𝟭 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧
🔬 𝟬.𝟬𝟮𝟴𝟴𝟭𝟳 𝗠𝗘𝗔𝗡 𝗞𝗟 𝗗𝗜𝗩𝗘𝗥𝗚𝗘𝗡𝗖𝗘
💾 𝟵𝟮.𝟵 𝗚𝗶𝗕 𝗚𝗣𝗨-𝗥𝗘𝗦𝗜𝗗𝗘𝗡𝗧 𝗪𝗘𝗜𝗚𝗛𝗧𝗦
I just finished running @TencentHunyuan’s newly released Hy4-Flash through my latest SAGE-REXL3 quantization process!
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗔 𝟮𝟮𝟬𝗕 𝗣𝗔𝗥𝗔𝗠𝗘𝗧𝗘𝗥 𝗠𝗢𝗘! 🧠
⚡ ~12B active parameters per token
🧠 Native 1M context architecture
🚀 Built-in 5-layer DFly2 speculative drafter
💻 Hybrid sparse/sliding-window attention
📦 4.84 bpw mixed-precision EXL3
And here’s the interesting part…
I preserved critical attention, routing and normalization components at higher precision while compressing the rest with SAGE.
The result?
𝗔 𝟵𝟲.𝟵𝟮% 𝗡𝗘𝗫𝗧-𝗧𝗢𝗞𝗘𝗡 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧 𝗪𝗜𝗧𝗛 𝗧𝗘𝗡𝗖𝗘𝗡𝗧’𝗦 𝗢𝗥𝗜𝗚𝗜𝗡𝗔𝗟 𝗕𝗙𝟭𝟲 𝗖𝗛𝗘𝗖𝗞𝗣𝗢𝗜𝗡𝗧! 🔥
━━━━━━━━━━━━━━━━━━
🖥️ 𝗗𝗘𝗦𝗜𝗚𝗡𝗘𝗗 𝗙𝗢𝗥 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞!
The active model weights occupy ~92.9 GiB, while the additional 60.24 GiB Engram lookup tables are stored in NVMe-backed memory-mapped files.
That’s the memory layout I’m targeting for single-Spark inference. Full serving and context validation are still ahead.
𝗔𝗡𝗗 𝗬𝗘𝗦… 𝗔 𝗙𝗨𝗟𝗟 𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗕𝗟𝗘 𝗜𝗡𝗙𝗘𝗥𝗘𝗡𝗖𝗘 𝗥𝗘𝗖𝗜𝗣𝗘 𝗜𝗦 𝗢𝗡 𝗜𝗧𝗦 𝗪𝗔𝗬! 👀🔥
🤗 𝗠𝗢𝗗𝗘𝗟 𝗪𝗘𝗜𝗚𝗛𝗧𝗦:
huggingface.co/vcruz305/Hy4-…
Huge credit to @TencentHunyuan and @tencentcloud for bringing Hy4-Flash to the open-weight community! 🙌
𝗛𝗬𝟰-𝗙𝗟𝗔𝗦𝗛. 𝟰.𝟴𝟰 𝗕𝗣𝗪. 𝟵𝟲.𝟵𝟮% 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.
𝗧𝗛𝗘 𝗦𝗔𝗚𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗝𝗢𝗨𝗥𝗡𝗘𝗬 𝗖𝗢𝗡𝗧𝗜𝗡𝗨𝗘𝗦! 🚀🔥
Hy4 Flash is now released on huggingface!! 🚀🚀🚀
Thanks to @aitrackerbot always being on point! You can finally run Haiku 5.5 locally 😎
Here’s what it had to say about the new model:
“Tencent just put the model from Monday's deleted announcement on Hugging Face:
• 220B total parameters, but only ~12B active per token, so it's cheap to run
• Reads text and images, with a 1M-token context window
• In Tencent's blind test with its own users, it scored 2.79 out of 4 vs 2.81 for Claude Haiku 5.5, at the lowest cost per task of any model tested
• About 1/20 the cost per task of full Hy4
• Also out: an FP8 version and the pre-trained base model
Definitely going to SAGE this one!! Stay tuned 👀
huggingface.co/tencent/Hy4-f…
Tencent may be cooking with this one...
Hy4 Flash on a single DGX Spark: Ripple 65B quality at nearly double the speed 🚀
I've spent the day putting Tencent's Hy4 Flash through its paces on one Spark, against Ripple 65B and the two Qwen3.8 open weight models.
Quality
Ripple 65B set the new standard on coding quality on a single Nvidia DGX Spark. Hy4 scored 76% on my 75-task reasoning eval, one point behind Ripple's 77% and well ahead of Qwen3.8 Flash-Next (63%) and 27B (56%).
Efficiency
This is the real killer feature - it gets there in about half the tokens, 11.7k per reply vs 22.6k for Ripple and roughly 25k for the Qwen models. Shorter replies plus faster decode means answers land much sooner.
Speed
up to 47.8 tok/s decode on code and 32 to 34 tok/s on prose, with prefill holding 846 tok/s at 2k and still 639 tok/s at 128k.
How it fits
Only 11.3B of its 155.5B parameters fire per token. The 63.5B n-gram memory is offloaded to SSD rather than sitting in VRAM, so the whole thing runs on one Spark with 295K context.
i'm 100% convinced...
that uncensored AI is the "LINUX" moment for AI.
it runs with or without a GPU, refuses nothing, and your data never leaves your machine.
here's how to build and run your OWN in 10 minutes ↓
Ivan Rocha retweeted
Gemini 4 Neon: frontier-level intelligence with fast end-to-end task completion across complex software engineering, knowledge work, and science workflows.
Begins rolling out today in the Gemini API in @GoogleAIStudio, Gemini for business, for paid Google AI Pro and Ultra subscribers in Google @Antigravity, and for Google AI Pro and Ultra subscribers in the @GeminiApp.
A free online book: LLM Inference Handbook by Sherlock Xu (2026).
This book is a practical technical reference for engineers deploying, scaling, and operating language models in production.
It brings together the concepts and decisions involved in turning a trained model into a responsive, cost-aware service, from model behavior and hardware selection to application interfaces and ongoing operations.
Readers can use it as a guided progression or as a lookup resource for specific engineering questions.
Read the book online on @ChapterPal: chapterpal.com/book/3fef12a8…
Ivan Rocha retweeted
🔄 You should update your vLLM ASAP. Lots of Local AI improvements.
This new release fixes a bug that left up to 84% of the available KV cache unusable on a DGX Spark.
And the new vLLM v0.34.0 release has some other serious Local AI optimizations hidden in the changelog.
Let's start with the Spark.
Running Qwen3.8-27B NVFP4 on 1x DGX Spark, developers discovered vLLM's new extensible KV cache was incorrectly accounting for reclaimable system memory.
Look at the results
💾 Before fix → 11.2 GiB KV cache
💾 After fix → 67.6 GiB KV cache 👈
That's 6x the usable KV-cache capacity! 🤯
Reported token-cache capacity
🔴 Before → 42,130 tokens
🟢 After → 253,952 tokens👈
vLLM also added Engram disk offloading for Qwen3.8-Flash-Next.
On 1x DGX Spark
💾 Loaded weights → 99.09 GiB → 72.09 GiB 👈
That's 27 GiB of memory saved.
And decode speed?
⚡ Before → 33.63 tps
⚡ After → 32.45 tps
Only ~3.5% slower
The trick? Keep the huge Engram lookup tables on NVMe and retrieve the necessary rows on demand.
Sound familiar?
I just posted about similar techniques for Tencent's 224B Hy4-Flash.
And AMD gets some ♥️ too.
On Strix Halo, an optional W4A16 optimization cut time-to-first-token
🧠 Qwen3-4B → 782ms to 578ms
🧠 Qwen3-8B → 1,328ms to 1,033ms
🧠 Gemma 4 31B → 8.75s to 6.88s
21–26% reduction
And 2x DGX Sparks running GLM-5.3-Flash also improved from 15.3 to 18.0 tps without speculation.
Ivan Rocha retweeted
huihui just updated their huge uncensored qwen3.8 27b repo 😳
its free, apache 2.0, no gate, 4.3m downloads
- new patch-all-5 file works on official unsloth ggufs from nov 11
- covers quants from iq1 to bf16
- same ungated repo, no signup
try it here: huggingface.co/huihui-ai/Hui…
Ivan Rocha retweeted
Until we get Qwhen4 😴, here's an update for Qwen3.8-Flash-Next for one DGX Spark ✨
The update is mostly powered by great community PRs!
- Updated to TensorFold 1.2.7 which increase decode speeds abit, and lower TTFT on 5 streams.
- Added VIDEO=1, so the model can handle video.
- Added PDF=1, so the model can read PDFs.
- Removed unneeded extra stuff from the base image.
Thanks for all the contributors!
Get it here:
github.com/MiaAI-Lab/Qwen3.8…
Ivan Rocha retweeted
huge: glm-5.3 flash on one spark running at 40 tok/s with 262k context.
int4 kv, dflash2 speculative decoding, persistent ssd cache, and a turn-tail patch that cuts first-token wait 37%.
recipe: github.com/sxuff/glm53-flash…
Ivan Rocha retweeted
INCREDIBLE
The BIBLE for understanding LLMs from first principles is now available online to read for free
Covers the model mechanics
- Tokens / tokenizers
- Transformers
- Attention
- KV cache
- Prefill vs decode
- Decoding controls
- Model packages
- Chat templates
- Long context
- RAG
- Agents / tools
- Fine-tuning
- Multimodal models
Then connects that to running models locally
- What "local" really means
- Open-weight vs opensource
- Quantization
- VRAM math
- Hardware tiers
- File formats / load safety
- Runtimes / serving modes
- Model selection
- Privacy
- Failure modes
- Benchmarks
- Practical setup paths
You should read this, and if you cannot now then you most definitely wanna bookmark it for later
Opensource AI FTW
Ivan Rocha retweeted
NVIDIA just released a speculative decoding draft on Hugging Face
It accelerates Qwen3.8-27B with 4B parameters, boosting token acceptance by 46% over the built-in head.
huggingface.co/nvidia/Qwen3.…
Zero-trust sandboxing for millions of agents on Kubernetes
github.com/agent-substrate/s…
1
49
if you have a dgx spark, give qwen3.8 27b at 6.0 bpw a try.
it's the closest thing to the full bf16 model that still fits on one spark. the one to pick when quality matters more than speed.
try it and tell me how it runs on your box. link in the first comment.
Ivan Rocha retweeted
A few more improvements to our LLM inference engine on AMD Strix Halo!
Qwen3.8-Flash-Next now reaches 1,885 tok/s in exact prefill at 8K on a single node, or 2,008 tok/s with our FAST profile.
DeepSeek V4.1 Flash is moving forward too. On two Strix Halo nodes over USB4, we're getting 1,165 tok/s exact and up to 1,425 tok/s FAST with longer prompts.
We've also been working on reducing GPU cycles per token, improving scheduling and making multi-request serving more efficient.
Some changes are already in main, while others are still being tested.
It's a lot of small improvements rather than one big breakthrough, but they keep adding up.
Ivan Rocha retweeted
DGX Spark, Qwen3.8-27B: median first token for 16 agents falls from 142 s to 25 s, answers token-identical
1 Spark, 128 GB: 16 agents, one chat of about 17,000 tokens each, all follow-ups at once
TensorFold 1.2.5
First token, median: 142.3 s
Speed of all replies: 27.4 tok/s
Chats found in cache: 0 of 16
TensorFold 1.2.7
First token, median: 25.4 s
Speed of all replies: 99.0 tok/s
Chats found in cache: 16 of 16
Ash's numbers on a GB10, from the 1.2.6 release notes: median first token 105 s to 25.8 s, speed 33.6 to 100.8 tok/s
Setup: official TensorFold 1.2.7 package, Qwen3.8-27B MLX 4-bit with DFlash2, default flags
Method: median over the 16 agents, 1.2.7 median of 3 runs (27.6 / 23.3 / 25.4 s), 1.2.5 one run
Method: temperature 0, thinking off, replies up to 512 tokens (our choice)
Video: replay of one recorded run at 12x with real timestamps, 1.2.7 median there 28.5 s
Ash's test tool is not published, so we rebuilt the load from his PR description. Our 142.3 s for 1.2.5 is our own value, not his 105 s.
TensorFold and the 1.2.6 cache work by Ash @ashxhart, DFlash2 draft by Inco AI @inco_ai, base model by Qwen @Alibaba_Qwen.
































