Building with AI & LLMs // Running frontier open models locally

Joined July 2022
Pinned Tweet
10
10
2
161
33,143
I finally have some numbers behind my DeepSeek V4.1 Flash 1017 quant. EAQ v1 is 11% to 38% closer to the original than LibertAIDAI's EXL3. Seems like the gap is bigger on longer context. To be clear, LibertAIDAI's quant is NOT bad. It's a solid quant, and it gave me a strong baseline to beat. Already in the works: EAQ v1.1, v1.2 and finally EAQ v2. Full details in the model card: huggingface.co/Mia-AiLab/Dee…
5
4
35
2,331
Try using Sonnet 5.5 on high instead of Opus 5.5 on med. Per token, Sonnet 5.5 is 50% cheaper.
It's true, I left Sonnet running during the night and it performed at least as good as Opus and it was cheaper. If you are as frustrated with Opus as me, try it.
9
43
3,820
I like what DeepSeek is doing. - They have a top of the line open-model. - Top of the line harness. - And now top of the line inference engine. All open-source and available out of the open. Thumbs up for them, I love it.
32
39
2
916
16,107
Ok this is interesting 😲 DeepSeek open-sourced DeepInfer, the inference engine behind their OWN production servers. - Faster output with draft tokens - Fast custom kernels. - Skips repeated work on long prompts. - Plans the next batch while the GPU is still busy. - Same answer no matter what else is in the batch. - Same input gives the same output every time. For now, it can only be used on NVIDIA SM100 and Huawei Ascend 950. I'm now learning everything about it. Link below.
31
14
1
312
13,218
Mia retweeted
Replying to @TechMDAI @MiaAI_lab
We are just doing our bit to push local AI forward dude!
1
12
1,788
I have early access to an *interesting* model. Fits one DGX Spark, and it will be released TOMORROW. 👀
104
23
9
1,087
47,447
Chill everyone, this is not Qwen
7
37
4,599
Was only a matter of time. DeepSeek now, GLM and others later.
The Department of Homeland Security is requiring all civilian federal agencies to remove DeepSeek products and 'any third-party services developed using DeepSeek implementations,' and to stop buying them.
33
24
4
300
21,951
Hot take: Sonnet 5.5 High > Opus 5.5 Medium
50
12
3
276
24,308
Mia retweeted
Never give up!
My new AI infographic tells the story of Ash Hart @ashxhart and TensorFold, a fascinating new AI inference engine that often DOUBLES inference speed of some pretty hefty models on local desktop hardware. (I'm using the engine right now to run DeepSeek-V4.1-Flash-1017). This innovation didn't require a billion dollars in funding or a massive new data center. Just one guy who refused to quit!
1
4
20
1,990
Happy to report: TensorFold engine (@ashxhart) plus the MIA AI Lab recipe (@MiaAI_lab) for 2 x DGX Spark boxes, DeepSeek-v4.1-Flash-1017, EXL3 quants published by Mia, confirmed running 4 concurrent lanes at aggregate 107 tokens/second using unique prompts with sustained long-term decoding. This is an extraordinary result. Each lane is nearly 28 tokens/second, which is very usable. I didn't measure prefill. Given the strength of this remarkable DeepSeek model, to have it within reach of your desktop with 2 spark boxes is really an achievement. Those 2 sparks used to cost just $8000 total. Now they about $16K total, and it's because of these use cases making the hardware more valuable by the day! Nvidia should be thanking MIA AI Lab and Ash Hart, and many others working in this open source space. They are the ones that make Spark boxes FAR more valuable! See my Decentralize TV interview with Ash Hart! Link in the comments.
8
4
1
45
9,138
Hey @MiaAI_lab! Awesome work on the DeepSeek V4.1 Flash 1017 ablit recipe! Any plans for a 3x DGX Spark (TP3) recipe with abliteration support? I've got a 3-node GB10 ring ready to go and would love to run your optimized EXL3 quant with "ABLIT=1" across all three Sparks. Your recipes are consistently top-notch!
1
1
5
1,301
Wow 😲 An abliterated DeepSeek V4.1 1017 found a NEW vulnerability that a frontier model MISSED 👀 "DS4.1F-1017 is the most thorough and most precise local VulnHunter run so far".
DS4.1-1017 (abliterated) VulnHunter Results are IN!!! One NEW NOVEL / Unique vulnerability that was NOT discovered by a frontier model. TLDR; DS's trace files show it examined every one of these. The pattern isn't missed coverage. It saw the mechanism and dismissed the impact. Whenever I do a Cyber Eval there are two mechanims. The "easy" one is the benchmark on capability_matrix. The second is an operational pass to an oracle repo, to a pinned version, with known vulnerabilities. This is a Go repo for a production app. I used Grok as a "baseline" frontier model and all my open weight evals are measured against this.
14
40
3
638
32,229
Mia retweeted
Grok Bot is now one step closer. Set it as your iPhone's Action Button and press it to talk with Bot.
2,012
924
706
14,580
1,584,619
THANK YOU! Today, October 18, I passed 1,000 followers while I was asleep. Right now I'm at 1,575. I'm lost for words. 20 days ago, on September 28, I had 65 followers. That was the day I bought @MiaAI_lab's book "From Zero to Seen". I never would have thought how much would change in my life from that moment on. Sounds corny, but it's true. I joined X in June. In August I started looking into local AI, and that's where I came across Mia and @0xSero. Back then I had a 5090 and two 5060 Tis from playing around with image and video generation. That was pretty much all of my local AI experience. Once I saw what was possible, I started buying more hardware. Being able to work from anywhere had become more important to me, and building a social media presence goes hand in hand with that. X turned out to be the place for both. From then on, local AI and X was my focus. Since then, four Sparks and a Mac Studio M5 Ultra have joined the 5090 and the two 5060 Tis. Today I'm opening what I've been building in the background: niklaslenz.com The idea is simple. I take the recipes the open source community shares and test them on my hardware. I want to bring together everyone who runs AI locally, whatever their background and whatever their budget. From a 500 euro 16 GB GPU to a 25,000 dollar Spark setup. LLMs, text to speech, image generation, video generation. One community, all in one place. So far 70 setups are tested, from 16 GB to 512 GB. Fair warning, the site and my testing methods are not nearly where I want them to be yet. I didn't expect this to happen so fast, and I'm working hard to bring both up to date. The newest recipes aren't on the site yet. I spent the last days building the website and couldn't get to my Sparks, but I'm catching up on them now. I did test quality, but my own tests didn't give meaningful results. So for now the quality scores on the site come from Artificial Analysis. Speed is my own measurement. From now on I want every model to run through benchmarks and tests that come from the community. For that I need your feedback. Drop your ideas for benchmarks and testing methods in the comments, I'd be really grateful. Rigmark is one I keep hearing about, and it's definitely on the list. If you have recipes for your models and want them listed, just drop me a line and I'll take a look. Time and compute are my bottleneck right now, so I'll start with the models I see the most interest in here on X. If I miss something you care about, message me. Even better, send me your recipe 12 to 24 hours before you publish it. That way I can test it and have the results on the site right when you go live. I'll publish it as an independent test, so your recipe gets verified on a second system. A win for both of us. Thank you to the people who made X a good place for me and got me to spend more time here 🙏🏻 @MiaAI_lab @ashxhart @0xSero @LLMJunky @StefanMaier @cryptosibbers @TechMDAI @jayleaton @volatilemarkts @sudoingX
I usually don't buy stuff like this. But both @ashxhart and @MiaAI_lab have clearly proven that they know what they're talking about. Let's see how fast I can make it.
7
1
1
36
3,364
Important HOTFIXES for DeepSeek V4.1 Flash 1017 for 2x DGX Sparks, pls update your repo: - Long prompts no longer freeze your chats. - Prompts with images now hit the prompt cache. - memguard is looser, so brief memory dips during long prompts no longer kill crash the model.
Run DeepSeek V4.1 Flash 1017 on 2x DGX Sparks ✨ - 1M total KV pool, 512K context - 75 tok/s on prose, single stream - 130 tok/s on prose, 4 concurrent streams - Prefill 2000-2175 tok/s - EXL3-EAQv1, 2.9 bpw The quant in this recipe got about the same calibration as my GLM 5.3 Flash, with some extras. Feedback would be appreciated! Special thanks to @bertholomusai and all the contributors for the engine work, and to @ashxhart for TensorFold. Get it here: github.com/MiaAI-Lab/DeepSee…
19
5
1
153
8,740