McSlash retweeted
Qwen3.8-27B has reportedly been compressed by 89.4%, going from a 55.6GB checkpoint to just 5.9GB while retaining 98.2% of its original aggregate benchmark performance.
The system behind this is Bonsai 2, the second generation of PrismML’s model compression technology.
What’s interesting is that this isn’t the first attempt.
The previous Bonsai 27B release reportedly preserved approximately 95% of the original model’s benchmark performance. Bonsai 2 raises that figure to 98.2% across 20 benchmarks.
That’s a 3.2 percentage-point improvement while keeping the compressed model remarkably small.
Here are the reported figures:
Original Qwen3.8-27B: 55.6GB
Bonsai 2 checkpoint: 5.9GB
Storage reduction: 89.4%
Benchmark performance retained: 98.2%
Previous Bonsai version: Approximately 95%
Evaluation coverage: 20 benchmarks
To understand why this is interesting, we need to look at how the compression works.
1. Ternary quantisation reduces the model to three weight values
Bonsai 2 uses a technique called ternary quantisation.
Most neural networks store their weights using numerical formats such as FP16, BF16, INT8 or INT4. These formats allow weights to represent different numerical values at varying levels of precision.
Ternary quantisation takes a much more aggressive approach.
Instead of representing weights using a broad range of numbers, it reduces them to just three possible values:
-1
0
+1
This significantly reduces the amount of information needed to represent individual weights.
For perspective, an FP16 weight uses 16 bits of storage, while a basic ternary representation theoretically needs considerably less than 2 bits per weight.
However, reducing the precision of billions of weights creates a difficult problem.
A model’s weights encode numerical relationships learned during training. Those relationships influence how it understands language, follows instructions, solves problems and generates code.
If too much information is discarded during compression, the model can lose important capabilities.
Bonsai 2 combines its compact ternary representations with scaling factors to preserve more of the useful information in the original weights.
The goal is to reduce storage requirements dramatically without causing an equally dramatic decline in model quality.
2. Bonsai 2 improves on the first generation
The difference between the two Bonsai releases is worth examining.
The original Bonsai 27B reportedly retained around 95% of Qwen3.8-27B’s aggregate benchmark performance.
Bonsai 2 increases this to 98.2% across 20 benchmarks.
That improvement suggests PrismML has made progress in preserving model quality despite using an extremely compact representation.
The company also highlights improvements in three particularly interesting areas:
Coding agents.
Multimodal reasoning.
Long-horizon tool use.
These capabilities matter because modern AI models are increasingly being used as components inside larger systems.
Consider a coding agent.
It needs to understand an existing repository, locate relevant files, modify code, execute commands, inspect errors and revise its implementation when something goes wrong.
This requires more than producing syntactically correct code. The model must maintain context, make decisions and interact with external tools across multiple steps.
If Bonsai 2 can preserve these capabilities in a 5.9GB checkpoint, it could become an interesting option for developers building local AI agents.
However, the aggregate benchmark score doesn’t establish that long-running agent workflows perform almost identically to the original. Those capabilities need dedicated evaluations.
3. What does reducing a 27B model to 5.9GB mean for local inference?
The original 55.6GB checkpoint creates a significant storage requirement.
Developers need enough disk space to download and maintain the model before even considering the hardware required to run it.
Bonsai 2 reduces that checkpoint to 5.9GB.
Qwen3.8-35B-A3B Runs at 50 Tokens/s on an RTX 3060 With 12GB VRAM
A 35-billion-parameter model generating around 50 tokens/s on an RTX 3060 with just 12GB of VRAM.
That’s the reported performance of Qwen3.8-35B-A3B running locally through llama.cpp, with an additional 16GB of system RAM.
The setup also reaches approximately 550 tokens/s during prompt processing and supports a configured context window of up to 262K tokens.
No RTX 4090. No multi-GPU configuration. Just a consumer GPU, system RAM and a carefully optimised inference setup.
The interesting engineering is how the developer gets a model of this size running despite the GPU not having enough memory to hold all its weights.
Four techniques make this possible: four-bit quantisation, CPU offloading, Flash Attention and an 8-bit KV cache.
The Hardware and Performance
The reported configuration:
Model: Qwen3.8-35B-A3B
GPU: NVIDIA RTX 3060
VRAM: 12GB
System RAM: 16GB
Inference engine: llama.cpp
Quantisation: Q4_K_M
Decode speed: Approximately 50 tokens/s
Prefill speed: Approximately 550 tokens/s
Configured context: Up to 262K tokens
Attention: Flash Attention
KV cache: Q8
CPU offloading: 26 MoE layers
One important distinction: the 550 tokens/s figure measures prompt processing, not response generation.
Prefill is the stage where the model processes incoming text. Decode is the stage where it generates the response, token by token.
So, the reported performance is approximately 550 tokens/s for prefill and 50 tokens/s for generation. Both are useful, but they measure different workloads.
How Does a 35B Model Run on 12GB VRAM?
Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) architecture.
Although the model contains approximately 35 billion parameters, only around 3 billion are activated for each token.
Instead of using every parameter during every computation, an MoE model uses a routing mechanism to select specific expert networks for each token.
This reduces the amount of computation required compared with activating the entire model.
However, there’s an important distinction between active parameters and memory requirements.
Having approximately 3B active parameters doesn’t mean the model only needs memory for 3B parameters. The full collection of expert weights still needs to be stored somewhere because different experts may be selected as generation progresses.
That’s why the developer combines quantisation with CPU offloading rather than relying on the GPU alone.
Splitting the Workload Between GPU and CPU
The RTX 3060 provides 12GB of VRAM, while the machine has another 16GB of system RAM.
Instead of forcing the entire model into GPU memory, the developer uses llama.cpp to distribute execution across both processors.
According to the reported configuration, 26 MoE layers are offloaded to the CPU.
The GPU handles the layers assigned to it, while the CPU processes the offloaded layers using system memory.
This allows the system to work with model weights that exceed the GPU’s physical memory capacity.
There is a performance trade-off, though.
Moving data between system RAM and GPU memory introduces communication overhead through PCIe. The CPU, RAM bandwidth, PCIe connection and layer distribution can all influence inference speed.
Maintaining around 50 tokens/s with this arrangement is therefore the particularly interesting part of the experiment.
Four Optimisations Behind the Performance
Each technique addresses a different memory or inference bottleneck.
1. Q4_K_M quantisation
The model uses Q4_K_M, a four-bit quantisation format that reduces the storage and memory bandwidth required for model weights.
Instead of representing weights at higher numerical precision, quantisation uses fewer bits to store them.
This makes larger models more practical on consumer hardware.
The trade-off is potential accuracy loss, which depends on the model, quantisation method and task being evaluated.
2. Flash Attention
McSlash retweeted
A 5.7GB AI MODEL WITH DEEPSEEK REASONING + QWEN 9B? 😳
This community fine tune is seriously interesting
• Only 5.7GB
• Strong reasoning capabilities
• Designed for minimal refusals
• Free to download and run locally
• No subscription or access gate
Get the model :
huggingface.co/mradermacher/…
McSlash retweeted
Had a girl who was like this. If I went to bed id wake up she'd either be on top of me or next to me. The good old days. Enjoy it while it lasts boys.
McSlash retweeted
GTA 6 is obviously a beta male simulator
White guy slaves for a mexican girl
yeah, no thanks
McSlash retweeted
The chances that this is all a coincidence rather than a highly sophisticated cyberattack are literally 0.
>unlimited battery
>works offline
>actually only works offline
>no one from the internet can hack into it
>somewhat limited, but will never run out of storage if only ONE person is using it
>less than $10 for lifetime
>can be easily inherited without any institutional procedure
>no government/institution involved
>can store any password RISK FREE
what else do you want from a password manager?
McSlash retweeted
Stop paying for intelligence.
You can run locally a 35B coding model on CPU with Computer-use,
- Ornith-1.5-35B-A3B
- 28-30 tokens/s decode
- 5k-255 tokens/s prefill on CPU/integrated setup,
- No GPU
- Consumer laptop proving capable open models work without high-end GPUs or cloud subscriptions.
- github.com/sybil-solutions/l…
McSlash retweeted
My schizo senses are tingling around this push to remove Jensen Huang.
I don't know if the goal is to nationalize Nvidia like Intel or make Jensen the scapegoat for the bubble popping or what, but something strange is afoot.
McSlash retweeted
33 years. The guy built the company that runs the entire AI boom. Dragged it back from the edge of bankruptcy more than once.
And people really think this is about an affair?
You can be as rich and as powerful as you want, right up until they decide you're not.
Watching this happen to him is genuinely sad, not gonna lie.
JUST IN: Jensen Huang fired by Nvidia $NVDA, according to the Wall Street Journal
McSlash retweeted
This video came out a decade ago and till today it perfectly captures how men can tell if the woman is into them. Hope it clears things up for you.
成龙年轻的时候,非常不相信女人。
因为身边的兄弟,钱经常被女人给卷跑,所以他也一直提防着自己老婆。
他和林凤娇结婚后,每年居然只给她基本生活费,让她在美国照顾儿子。两人就这样过了将近20年。
直到1999年,出轨吴绮莉的事件爆发。
成龙当时做好了离婚的准备,可没想到林凤娇的反应却很大度,一句都没骂他,还说要站出来和他一起面对。
他大受感动,意识到林凤娇如此贤惠,才决定修改遗嘱,把巨额财产留给林凤娇和儿子。
吴绮莉这个事情,没想到却阴差阳错地帮林凤娇坐稳了龙嫂的位置,从那以后开始掌管成龙的财富。
看来女人的智慧也很重要啊,要是其它女人,结果想必完全不一样。
McSlash retweeted
❗ 폴란드 정부의 허위 정보 ❗
크라쿠프 테러범은 극우가 아니라 좌파였습니다!
페이스북 프로필이 차단되기 전에 직접 들어가 봤습니다.
프로필 내용으로 보면 오히려 공산주의와 LGBT+에 호의적이었던 것으로 보입니다. 기독교인들과 교황 요한 바오로 2세를 모욕하기도 했습니다.
McSlash retweeted
🚨🇮🇱 BREAKING — AIPAC WON!
AIPAC Won Most Seats in the House.
🇮🇱⚫ (231) — AIPAC Bloc
🇺🇸🔵 (164) — Democrats
🇺🇸🔴 (11) — Republicans
McSlash retweeted
ChatGPT has now a big problem.
Researchers at Oxford exposed a massive threat to large language models.”
It’s called "Model Collapse."
And it explains why the internet is about to become a toxic waste dump for Large Language Models.
Right now, AI companies are scraping the entire web to train their next-generation models.
But there’s a massive problem.
The internet is no longer just human data. It is rapidly filling up with AI-generated text, images, and code.
What happens when an AI trains on data generated by another AI?
A new paper in Nature just gave us the answer. And it is terrifying.
When AI models are recursively trained on AI-generated data, they suffer from irreversible brain damage.
The researchers found that the model starts forgetting the "tails" of the original data distribution. The rare, weird, unique, and complex human nuances completely disappear.
It's like making a photocopy of a photocopy.
Every generation loses resolution. The output gets flatter, blander, and more generic, until the model completely collapses in on itself.
For years, we thought the biggest bottleneck for AI was compute, getting enough H100 GPUs from Nvidia.
This paper proves the real bottleneck is us.
If we keep flooding the internet with synthetic AI slop, we will poison the very data pool required to build smarter models.
The ultimate irony? We thought AI was going to make human creators obsolete.
Instead, this research proves that verified, genuine human data is about to become the most valuable commodity in the tech industry.





































