McSlash (@Kaiothic) profile picture

Simple times

Joined February 2017
Photo by McSlash (@Kaiothic)
Qwen3.8-27B has reportedly been compressed by 89.4%, going from a 55.6GB checkpoint to just 5.9GB while retaining 98.2% of its original aggregate benchmark performance. The system behind this is Bonsai 2, the second generation of PrismML’s model compression technology. What’s interesting is that this isn’t the first attempt. The previous Bonsai 27B release reportedly preserved approximately 95% of the original model’s benchmark performance. Bonsai 2 raises that figure to 98.2% across 20 benchmarks. That’s a 3.2 percentage-point improvement while keeping the compressed model remarkably small. Here are the reported figures: Original Qwen3.8-27B: 55.6GB Bonsai 2 checkpoint: 5.9GB Storage reduction: 89.4% Benchmark performance retained: 98.2% Previous Bonsai version: Approximately 95% Evaluation coverage: 20 benchmarks To understand why this is interesting, we need to look at how the compression works. 1. Ternary quantisation reduces the model to three weight values Bonsai 2 uses a technique called ternary quantisation. Most neural networks store their weights using numerical formats such as FP16, BF16, INT8 or INT4. These formats allow weights to represent different numerical values at varying levels of precision. Ternary quantisation takes a much more aggressive approach. Instead of representing weights using a broad range of numbers, it reduces them to just three possible values: -1 0 +1 This significantly reduces the amount of information needed to represent individual weights. For perspective, an FP16 weight uses 16 bits of storage, while a basic ternary representation theoretically needs considerably less than 2 bits per weight. However, reducing the precision of billions of weights creates a difficult problem. A model’s weights encode numerical relationships learned during training. Those relationships influence how it understands language, follows instructions, solves problems and generates code. If too much information is discarded during compression, the model can lose important capabilities. Bonsai 2 combines its compact ternary representations with scaling factors to preserve more of the useful information in the original weights. The goal is to reduce storage requirements dramatically without causing an equally dramatic decline in model quality. 2. Bonsai 2 improves on the first generation The difference between the two Bonsai releases is worth examining. The original Bonsai 27B reportedly retained around 95% of Qwen3.8-27B’s aggregate benchmark performance. Bonsai 2 increases this to 98.2% across 20 benchmarks. That improvement suggests PrismML has made progress in preserving model quality despite using an extremely compact representation. The company also highlights improvements in three particularly interesting areas: Coding agents. Multimodal reasoning. Long-horizon tool use. These capabilities matter because modern AI models are increasingly being used as components inside larger systems. Consider a coding agent. It needs to understand an existing repository, locate relevant files, modify code, execute commands, inspect errors and revise its implementation when something goes wrong. This requires more than producing syntactically correct code. The model must maintain context, make decisions and interact with external tools across multiple steps. If Bonsai 2 can preserve these capabilities in a 5.9GB checkpoint, it could become an interesting option for developers building local AI agents. However, the aggregate benchmark score doesn’t establish that long-running agent workflows perform almost identically to the original. Those capabilities need dedicated evaluations. 3. What does reducing a 27B model to 5.9GB mean for local inference? The original 55.6GB checkpoint creates a significant storage requirement. Developers need enough disk space to download and maintain the model before even considering the hardware required to run it. Bonsai 2 reduces that checkpoint to 5.9GB.
Qwen3.8-35B-A3B Runs at 50 Tokens/s on an RTX 3060 With 12GB VRAM A 35-billion-parameter model generating around 50 tokens/s on an RTX 3060 with just 12GB of VRAM. That’s the reported performance of Qwen3.8-35B-A3B running locally through llama.cpp, with an additional 16GB of system RAM. The setup also reaches approximately 550 tokens/s during prompt processing and supports a configured context window of up to 262K tokens. No RTX 4090. No multi-GPU configuration. Just a consumer GPU, system RAM and a carefully optimised inference setup. The interesting engineering is how the developer gets a model of this size running despite the GPU not having enough memory to hold all its weights. Four techniques make this possible: four-bit quantisation, CPU offloading, Flash Attention and an 8-bit KV cache. The Hardware and Performance The reported configuration: Model: Qwen3.8-35B-A3B GPU: NVIDIA RTX 3060 VRAM: 12GB System RAM: 16GB Inference engine: llama.cpp Quantisation: Q4_K_M Decode speed: Approximately 50 tokens/s Prefill speed: Approximately 550 tokens/s Configured context: Up to 262K tokens Attention: Flash Attention KV cache: Q8 CPU offloading: 26 MoE layers One important distinction: the 550 tokens/s figure measures prompt processing, not response generation. Prefill is the stage where the model processes incoming text. Decode is the stage where it generates the response, token by token. So, the reported performance is approximately 550 tokens/s for prefill and 50 tokens/s for generation. Both are useful, but they measure different workloads. How Does a 35B Model Run on 12GB VRAM? Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) architecture. Although the model contains approximately 35 billion parameters, only around 3 billion are activated for each token. Instead of using every parameter during every computation, an MoE model uses a routing mechanism to select specific expert networks for each token. This reduces the amount of computation required compared with activating the entire model. However, there’s an important distinction between active parameters and memory requirements. Having approximately 3B active parameters doesn’t mean the model only needs memory for 3B parameters. The full collection of expert weights still needs to be stored somewhere because different experts may be selected as generation progresses. That’s why the developer combines quantisation with CPU offloading rather than relying on the GPU alone. Splitting the Workload Between GPU and CPU The RTX 3060 provides 12GB of VRAM, while the machine has another 16GB of system RAM. Instead of forcing the entire model into GPU memory, the developer uses llama.cpp to distribute execution across both processors. According to the reported configuration, 26 MoE layers are offloaded to the CPU. The GPU handles the layers assigned to it, while the CPU processes the offloaded layers using system memory. This allows the system to work with model weights that exceed the GPU’s physical memory capacity. There is a performance trade-off, though. Moving data between system RAM and GPU memory introduces communication overhead through PCIe. The CPU, RAM bandwidth, PCIe connection and layer distribution can all influence inference speed. Maintaining around 50 tokens/s with this arrangement is therefore the particularly interesting part of the experiment. Four Optimisations Behind the Performance Each technique addresses a different memory or inference bottleneck. 1. Q4_K_M quantisation The model uses Q4_K_M, a four-bit quantisation format that reduces the storage and memory bandwidth required for model weights. Instead of representing weights at higher numerical precision, quantisation uses fewer bits to store them. This makes larger models more practical on consumer hardware. The trade-off is potential accuracy loss, which depends on the model, quantisation method and task being evaluated. 2. Flash Attention
13
39
1
340
22,926
McSlash retweeted
A 5.7GB AI MODEL WITH DEEPSEEK REASONING + QWEN 9B? 😳 This community fine tune is seriously interesting • Only 5.7GB • Strong reasoning capabilities • Designed for minimal refusals • Free to download and run locally • No subscription or access gate Get the model : huggingface.co/mradermacher/…
6
16
137
8,933
Had a girl who was like this. If I went to bed id wake up she'd either be on top of me or next to me. The good old days. Enjoy it while it lasts boys.
There are women who like their man. Never black pill.
97
288
13
10,279
1,150,697
GTA 6 is obviously a beta male simulator White guy slaves for a mexican girl yeah, no thanks
75
237
7
3,629
32,123
The chances that this is all a coincidence rather than a highly sophisticated cyberattack are literally 0.
64
50
3
948
73,529
McSlash retweeted
>unlimited battery >works offline >actually only works offline >no one from the internet can hack into it >somewhat limited, but will never run out of storage if only ONE person is using it >less than $10 for lifetime >can be easily inherited without any institutional procedure >no government/institution involved >can store any password RISK FREE what else do you want from a password manager?
Password managers ARE SO COOKED $10 lifetime
44
331
9
4,396
197,636
McSlash retweeted
Claude and GPT are now FREE 😳 Duck AI just added the new Claude 5.5 models > Haiku 5.5 FREE for everyone > Sonnet 5.5 on Plus and Pro > Opus 5.5 on Pro GPT 6 Luna and gpt-oss 120B are FREE there too try it: duck.ai
27
43
1
487
42,649
Stop paying for intelligence. You can run locally a 35B coding model on CPU with Computer-use, - Ornith-1.5-35B-A3B - 28-30 tokens/s decode - 5k-255 tokens/s prefill on CPU/integrated setup, - No GPU - Consumer laptop proving capable open models work without high-end GPUs or cloud subscriptions. - github.com/sybil-solutions/l…
Ornith-1.5-35B-3A running locally on a 3,000$ Dell XPS laptop (64GB DDR5 with integrated graphics) It's editing a screen-recording in Pi + T3Code trimming out a tailscale url that leaked in my recording. It's running at 30 tok/s decode and 1.5k tok/s prefill. Omarchy Forever
11
16
4
220
13,165
아침 6시 반부터 밤 10시 반까지, 한중일 여고생의 하루 🇰🇷🇨🇳🇯🇵
🤖 Made with AI
455
342
133
5,388
2,906,985
My schizo senses are tingling around this push to remove Jensen Huang. I don't know if the goal is to nationalize Nvidia like Intel or make Jensen the scapegoat for the bubble popping or what, but something strange is afoot.
101
84
3
1,397
27,921
McSlash retweeted
33 years. The guy built the company that runs the entire AI boom. Dragged it back from the edge of bankruptcy more than once. And people really think this is about an affair? You can be as rich and as powerful as you want, right up until they decide you're not. Watching this happen to him is genuinely sad, not gonna lie.
JUST IN: Jensen Huang fired by Nvidia $NVDA, according to the Wall Street Journal
214
213
21
2,077
128,044
🇺🇸⚡️- BREAKING: Nvidia is preparing to oust Jensen Huang, according to The Wall Street Journal.
12
15
314
9,663
McSlash retweeted
This video came out a decade ago and till today it perfectly captures how men can tell if the woman is into them. Hope it clears things up for you.
Men please be honest. Are y'all really clueless when a woman is into you, or do you pretend like you don't know because the feeling isn't mutual?
471
11,583
628
158,323
4,322,887
McSlash retweeted
成龙年轻的时候,非常不相信女人。 因为身边的兄弟,钱经常被女人给卷跑,所以他也一直提防着自己老婆。 他和林凤娇结婚后,每年居然只给她基本生活费,让她在美国照顾儿子。两人就这样过了将近20年。 直到1999年,出轨吴绮莉的事件爆发。 成龙当时做好了离婚的准备,可没想到林凤娇的反应却很大度,一句都没骂他,还说要站出来和他一起面对。 他大受感动,意识到林凤娇如此贤惠,才决定修改遗嘱,把巨额财产留给林凤娇和儿子。 吴绮莉这个事情,没想到却阴差阳错地帮林凤娇坐稳了龙嫂的位置,从那以后开始掌管成龙的财富。 看来女人的智慧也很重要啊,要是其它女人,结果想必完全不一样。
102
109
10
2,334
499,889
❗ 폴란드 정부의 허위 정보 ❗ 크라쿠프 테러범은 극우가 아니라 좌파였습니다! 페이스북 프로필이 차단되기 전에 직접 들어가 봤습니다. 프로필 내용으로 보면 오히려 공산주의와 LGBT+에 호의적이었던 것으로 보입니다. 기독교인들과 교황 요한 바오로 2세를 모욕하기도 했습니다.
542
6,692
334
23,173
821,101
McSlash retweeted
🚨🇮🇱 BREAKING — AIPAC WON! AIPAC Won Most Seats in the House. 🇮🇱⚫ (231) — AIPAC Bloc 🇺🇸🔵 (164) — Democrats 🇺🇸🔴 (11) — Republicans
600
4,867
591
33,437
2,730,992
McSlash retweeted
🚨🇺🇸 BREAKING: AIPAC has dissolved publicly and changed their name to UNITED DEMOCRACY PROJECT This is their new spending strategy.
371
5,721
215
13,153
128,841
McSlash retweeted
Replying to @Jvnior
"Democracy" simply means a nation that is run by J**S. American Poet Ezra Pound
1
2
21
1,796
McSlash retweeted
ChatGPT has now a big problem. Researchers at Oxford exposed a massive threat to large language models.” It’s called "Model Collapse." And it explains why the internet is about to become a toxic waste dump for Large Language Models. Right now, AI companies are scraping the entire web to train their next-generation models. But there’s a massive problem. The internet is no longer just human data. It is rapidly filling up with AI-generated text, images, and code. What happens when an AI trains on data generated by another AI? A new paper in Nature just gave us the answer. And it is terrifying. When AI models are recursively trained on AI-generated data, they suffer from irreversible brain damage. The researchers found that the model starts forgetting the "tails" of the original data distribution. The rare, weird, unique, and complex human nuances completely disappear. It's like making a photocopy of a photocopy. Every generation loses resolution. The output gets flatter, blander, and more generic, until the model completely collapses in on itself. For years, we thought the biggest bottleneck for AI was compute, getting enough H100 GPUs from Nvidia. This paper proves the real bottleneck is us. If we keep flooding the internet with synthetic AI slop, we will poison the very data pool required to build smarter models. The ultimate irony? We thought AI was going to make human creators obsolete. Instead, this research proves that verified, genuine human data is about to become the most valuable commodity in the tech industry.
433
1,331
457
6,068
966,552
🇮🇱⚡️- Opposition leader Gadi Eisenkot has likely secured the first mandate to form a government, receiving 56 recommendations compared to Netanyahu’s 46. The Arab party Ra’am has endorsed Eisenkot to form a government.
2
18
2
249
12,910