Qwen3.8-35B-A3B is reportedly generating around 50 tokens/sec on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The setup also reaches approximately 550 tokens/sec in prompt processing, with a configured context window of 262K tokens. Reported configuration: GPU: RTX 3060 12GB System RAM: 16GB Quantization: Q4_K_M Generation: ~50 tokens/sec Prompt processing: ~550 tokens/sec Context: 262K tokens Runtime: llama.cpp with Flash Attention and Q8 KV cache How does a 35B model run on this hardware? The secret is its Mixture-of-Experts (MoE) architecture. Qwen3.8-35B-A3B contains around 35B parameters but activates approximately 3B per token. That reduces computation during generation. However, all model weights still need to reside in memory. The reported setup works around the 12GB VRAM limit by offloading 26 MoE layers to the CPU. This GPU-CPU split makes local inference possible, although performance depends on memory bandwidth, offloading overhead and context length. The next test is agentic coding. The developer plans to compare it against Ornith-1.5-35B, Qwen3.8-27B and the base Qwen3.6-35B-A3B in Hermes and Pi coding-agent workflows. The question isn’t just whether the model generates code quickly. Can it use tools correctly, debug errors and finish multi-step projects without constant human intervention? Also, a configured 262K context window doesn’t guarantee the same speed at maximum context. If the model performs well in these tests, it could offer developers another option for running local coding agents on affordable hardware. The speed numbers are promising. Reliable task completion will determine how useful this setup really is.

Bonsai 2 27B reportedly compresses Qwen3.8-27B from 55.6GB to just 5.9GB while retaining 98.2% of its aggregate performance across 20 benchmarks. That’s a model more than 9x smaller, with a reported performance retention of 98.2%. The comparison: Qwen3.8-27B: 55.6GB Bonsai 2 27B: 5.9GB Reported performance retention: 98.2% Previous Bonsai generation: Approximately 95% retention The technology behind this is ternary quantization. Instead of storing model weights using conventional higher-precision numerical formats, ternary quantization represents them using an extremely limited set of values. This dramatically reduces the storage required for the model. But compressing a model is the easy part to describe. Preserving its capabilities is the real challenge. According to PrismML, Bonsai 2 improves on the previous generation in areas such as coding agents, multimodal reasoning, and long-horizon tool use. These tasks are particularly demanding because an AI agent must do more than produce a good answer. It needs to plan, execute tools, interpret their outputs, recover from mistakes, and maintain context across multiple steps. If a compressed model can preserve those abilities, the hardware implications are significant. A 27B model that fits into 5.9GB could become accessible to more users with consumer GPUs, laptops, and edge devices. It could also reduce memory requirements in larger inference systems, potentially allowing more models or concurrent workloads to run on the same hardware. There is one important caveat: 98.2% aggregate benchmark retention does not mean 98.2% accuracy on every task. Results depend on the benchmarks selected, their weighting, and the evaluation methodology. Independent testing is needed to establish how well that performance translates to real-world workloads. Still, the reported progression is worth watching. The original Bonsai 27B reportedly retained around 95% of its parent model’s benchmark performance. Bonsai 2 raises that figure to 98.2%, while reducing the model’s footprint to 5.9GB. 55.6GB → 5.9GB. More than 9x smaller. 98.2% reported performance retention. If independent evaluations confirm these results, ternary quantization could make capable large models considerably more practical to run locally.

Qwen3.8-27B reportedly went from 82 tokens/sec to 381 tokens/sec on a single RTX 3090 with 24GB VRAM, without changing the GPU. The entire improvement came from optimizing the inference stack. Here’s how the developer reached that speed: • Initial setup: ~82 tokens/sec • Optimized MTP: ~114 tokens/sec • DFlash2 + context lookup: ~138 tokens/sec • Optimized lookup verification: ~381 tokens/sec The final result is roughly 4.6× the initial generation speed. But the most interesting part is how they achieved it. 1. DFlash2 reduces sequential generation Normally, an autoregressive model generates tokens one at a time, with each step depending on the previous output. Speculative decoding changes this. A smaller draft mechanism proposes multiple tokens, and the main model verifies them together. When the proposals are accepted, the system avoids some of the sequential work required by ordinary decoding. DFlash2 builds on this approach to accelerate token generation. 2. Context lookup reuses text that already exists The developer added another optimization: searching the input context for text that matches the expected output. This is particularly useful when a model needs to reproduce content from a document, rather than generate entirely new text. The optimized system can propose matching sequences from the existing context and ask the main model to verify them. Instead of verifying the usual seven proposed tokens, the setup can verify blocks of up to 16. The developer reports acceptance of approximately 15 out of 16 tokens per verification step. That means far fewer sequential decoding steps are needed for this particular workload. The benchmark shows why the distinction matters. On a document-reproduction task involving approximately 25K tokens: • Previous DFlash2 setup: ~260 tokens/sec • Optimized context lookup: ~382 tokens/sec However, 381 tokens/sec is not the model’s general-purpose chat speed. For ordinary conversational workloads, the developer reports around 133 tokens/sec. Context lookup works best when the output overlaps heavily with text already available in the prompt. It won’t deliver the same gains when generating original prose or solving problems that require entirely new output. Potential applications include document processing, retrieval-augmented generation, code transformations and other workflows that reuse existing text. Prefix caching delivers another major improvement. When asking a second question about the same 25K-token document, the system can reuse the computation from the shared prompt prefix. Reported time to first token dropped from 22.4 seconds to just 0.56 seconds. That’s approximately 40× faster for the repeated-prefix scenario. The model still has to process the new question and generate its answer, but it doesn’t need to recompute the entire shared prefix from scratch. The broader lesson is that local inference performance depends on more than model size and GPU specifications. Speculative decoding reduces sequential generation. Context lookup exploits repeated text. Prefix caching avoids redundant computation. Combined, these techniques can produce enormous gains when the workload is a good fit. Code: syv-ai/qwen38-27b-rtx3090 on GitHub.