Pinned Tweet
I compared Strata engines 0.1.52 and 0.1.59 on my Windows laptop with an RTX 4090 Laptop GPU (16 GB VRAM) and 80 GB of RAM. The biggest gain was in prefill, the time spent processing the prompt before the model starts writing.
Both builds ran Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS with a 524,288-token context, YaRN ×2, int8 KV cache with 32,768 tokens resident on the GPU, MTP window 4 and a 2,304 MiB VRAM reserve. The Python server moved from Strata 0.2.6 to 0.2.13.
Fresh prompts · engine 0.1.52 → 0.1.59
• 8,473 input tokens, 3 runs per build. Prefill 2,133 → 2,430 tok/s. First token 4.02 → 3.52 s. Decode 104.5 → 103.5 tok/s.
• 134,568 input tokens, 2 runs per build. Prefill 2,544 → 2,864 tok/s. First token 53.07 → 47.14 s. Decode 94.4 → 94.3 tok/s.
• 453,109 input tokens, 1 run per build. Prefill 2,217 → 2,431 tok/s. First token 204.66 → 186.69 s. Decode 85.5 → 87.8 tok/s.
Each fresh test reused zero input tokens and generated 256 tokens with greedy decoding. Repeated runs are reported as medians.
I also ran five matched cached follow-ups per build at each size. At 8.5K, median decode was 92.7 → 95.3 tok/s. At 135K it was 88.7 → 87.5 tok/s, with time to first token around 1.9 s on both builds. Each build reread a median of 61 input tokens in those cached tests.
A separate MTP test at temperature 0.6 gave mixed results. Moving from window 4 to 6 raised median decode from 68.2 to 73.3 tok/s at 399K and lowered it from 81.6 to 79.5 tok/s at 135K. I kept window 4 for daily use.
Thanks to @coldniko for building Strata. Engine link in the first comment.
#LocalAI #Strata #Qwen
The previous Qwen Flash was already impressive for its speed and efficiency.
But what I'm really hoping for with Qwen4-Flash is a much bigger leap in reasoning, coding and agentic capabilities without sacrificing that speed.
A Flash model that can think deeper while staying fast would be a serious upgrade for local AI agents.
Really looking forward to this one.
Finally some good news !!
About Qwen4 ofcourse !
Qwen dev's have been improving Qwen4 so much, thanks to RL that the version we are going to get is a vastly improved version, basically a version bumb or aka Qwen4.1.
This is really good, they know they can't release an inferior model, pressure is huge.
Now my hopes for Qwen4-27B is even higher !!
LFG !!!!
Only question is when, but they can take a bit more time if it means we get something that meets our expectations !
3
163
Used GPUs are the quiet superpower of local AI. A card one generation old runs the same quantized model for a fraction of the price, you just trade a little speed for it. If you're running local models on a used card, which one?
3
47
Been looking at Linger + PolyStrata.
The idea is interesting: run a local AI agent that can dynamically load and swap models instead of keeping everything loaded in memory.
I'm putting it through its paces on my RTX 4090 Laptop (16GB VRAM + 80GB RAM).
Want to see how far it can go and what the speed trade-offs look like.
Benchmark coming soon. 👀
4
4
253
Linger: github.com/simplexai-labs/li…
PolyStrata: github.com/simplexai-labs/Po…
3
43
gputaskman answers what Task Manager leaves open: not how much GPU memory is used, but by what. It ties per-process VRAM and engine activity to the runtime that holds it, and separates model weights from KV cache using GGUF metadata. Windows, Linux and WSL across NVIDIA, AMD and Intel, MIT-licensed and on PyPI. The project states plainly that device and process counters are measurements while model identification and buffer breakdowns are estimates, and calls it early software. github.com/shanselman/gputas…
1
3
107
I compared my daily Strata setup with BigMoeOnEdge 0.46.0 on the same Qwen3.8-Flash-Next IQ3_XXS weights. The machine is a Windows laptop with an RTX 4090 Laptop GPU. This is the official Windows build with its CUDA pack, and the run used the GPU. The Strata figures are yesterday's corrected measurements.
Both sides read a fresh prompt and wrote 1,024 tokens at temperature 0.6, with thinking off. BigMoe medians are four fresh runs. Strata medians are eight fresh runs across two loads. BigMoe counted 7,896 to 7,902 input tokens. Strata counted 7,876, except one load at 7,877:
• Prefill, reading the prompt: Strata 1,926 input tokens/s, BigMoe 609. • First token: 4.15 s and 13.12 s. • Decode, writing the answer: Strata 72.0 output tokens/s, BigMoe 15.4. At about 32,100 input tokens, two runs each, Strata still read and wrote faster: 2,181 versus 574 input tokens/s, and the first token arrived in 14.80 s versus 56.14 s. Decode was 70.3 versus 13.9 output tokens/s. Strata was on its daily settings, including MTP window 4. BigMoe's automatic plan left streaming off, and this file has no MTP head, so drafting stayed off. Where BigMoe is aimed is a model that does not fit one device: the experts used most stay in GPU memory, the rest in RAM, and the drive after that. It can also split one model across two GPUs in the same computer. This laptop has one GPU, and the plan kept the weights in memory, so this test did not measure that path. BigMoe found the three planted codes and passed the same four small cache tests. That checks that the run was real, not that the engines match everywhere.
BigMoe wrote the reply at 15.4 tokens/s and Strata at 72.0. So I'm still gonna use Strata as my daily driver
#LocalAI #Strata #Qwen #BigMoeOnEdge
2
6
393
Model, IQ3_XXS:
huggingface.co/ISTA-DASLab/Q…
BigMoeOnEdge 0.46.0:
github.com/Helldez/BigMoeOnE…
Strata: github.com/Niko1221/Strata/r…
1
67
Imagine letting an AI buy and sell crypto for you.
You give it a $1,000 budget and tell it never to accept a trade with more than 2% price slippage.
But what if the price changes after the AI approves the trade?
Proof-Gated Signing adds a security layer to the wallet itself. It checks the actual transaction before allowing your money to move.
In its research tests, it blocked 131 out of 140 harmful scenarios.
A promising approach to making autonomous crypto agents safer.
#AI #Crypto #Web3
2
6
225
Research + source code:
GitHub: github.com/LoopGlitch26/Proo…
Paper: arxiv.org/abs/2610.00354
Technical breakdown: cygnuxlabs.com/research/guar…
The important distinction: this is a research prototype tested against scripted attacks, not a production wallet or independently reproduced mainnet result.
1
52
I compared my daily Strata setup with a local experimental Basalt port on the same Qwen3.8-Flash-Next IQ3_XXS weights. The machine is a Windows laptop with an RTX 4090 Laptop GPU. This is not an official Basalt result. Official Basalt targets Blackwell GPUs.
Both sides read a fresh prompt and wrote 1,024 tokens at temperature 0.6, with thinking off. The medians below use eight runs across two loads. The input was 7,876 tokens, except one Strata load at 7,877:
• Prefill, reading the prompt: Strata 1,926 input tokens/s, Basalt 1,979. • First token: 4.15 s and 4.05 s. • Decode, writing the answer: Strata 72.0 output tokens/s, Basalt 33.9.
At 32,103 input tokens, two runs in the later load, Basalt read the prompt faster: 3,022 versus 2,181 input tokens/s, and the first token arrived in 10.70 s versus 14.80 s. Decode was still 35.4 versus 70.3 output tokens/s.
Strata was on its daily settings, including MTP window 4. The Basalt pack has no drafter, and the conversion is not a bit-identical copy of the GGUF. Both found three planted codes and passed the same four small cache tests. That checks that the runs were real, not that the models match everywhere.
Basalt wrote the reply at 33.9 tokens/s and Strata at 72.0. I’m going to keep using Strata.
#LocalAI #Strata #Qwen
4
16
1,612
Strata 0.1.59 with the 0.2.13 server, daily IQ3_XXS config: MTP window 4 at confidence 0.5, prefill auto, 524,288 context capacity, INT8 KV with 32,768 positions resident, YaRN ×2, routing bias 1.0, and the binary's default tail-skip of 7. The 8K median uses every fresh run. One of the two Strata loads counted 7,877 input tokens instead of 7,876. Ranges: Strata prefill 1,709–1,960 input tokens/s, first token 4.07–4.68 s, decode 69.6–75.2 output tokens/s. Basalt prefill 1,852–2,089, first token 3.85–4.33 s, decode 30.0–36.3.
The Basalt build is a local Ada port of 0.4.1 commit 899e820. It used the host-planned expert path, prefill auto, a 40,960 context, and no drafts. The 32K pair was not alternated between engines. I could not run a clean drafting-off Strata control: this pack refuses to start without a speculative window of at least 2. On the eight daily runs, Strata accepted 511–556 of the 907–975 drafts it offered.
GPU power was sampled rather than taken from the driver's N/A limit. It reached about 174 W and 65 C during the session. The source files and the converted pack are on different drives.
Strata: github.com/Niko1221/Strata
Basalt: github.com/jesdga95/basalt
1
144
If you run llama.cpp on an Intel Panther Lake laptop (Core Ultra Series 3), today's b12503 cuts per-token host overhead in the OpenVINO backend: graph bindings are now reused between decode calls and the KV cache moves to a 4-D stateless layout that avoids full-cache copies each decode. Release notes report host overhead outside the main function falling from 0.5–1.1 ms to 0.14–0.38 ms per token. Caveat: the new KV layout can break attention on some models (e.g. Phi-3.5/Phi-4-mini); set GGML_OPENVINO_DISABLE_KV_4D=1 to revert.
700 followers! 🥳
Thank you to everyone who followed and supported BlueOrbit Lab! ❤️
If you're into AI, tech or building cool things, let's connect.
What are you working on right now? 👇
7
7
286
vLLM v0.34.0 changed how much of the GPU the engine assumes it owns. The KV cache is now sized from the memory actually free after warmup, and --gpu-memory-utilization defaults to 1.0. To keep the old sizing, pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.
The release gives its own trade-off: a bigger cache and more throughput at high concurrency, but per-token decode 6-18% slower on some models at concurrency 128. That is the release note's wording, not a run of mine. vLLM has no native Windows build.
Local Qwen3.8-Flash-Next tried painting MrBeast through Strata + Klad.
7m 14s, 70 brush strokes and 40 fills. It hit the 12-request cap. The likeness still needs work.
Actual edits replayed below, with waits condensed. Klad repo in the first reply.
Who should we paint next?
2
5
237
Klad: github.com/klad-app/klad
Reference photo: Steven Khan, CC BY 4.0, resized. Painted study shown alongside it.
commons.wikimedia.org/wiki/F…
creativecommons.org/licenses…
4
48
Imagine reading a 200-page technical manual and asking the website to explain a complicated section using the AI model running on your own PC, without sending the text to a cloud AI service.
LumaBrowser is experimenting with exactly this kind of integration.
Websites can request access to its local model, with permission required for each site.
It's early (text-only for now), but it opens an interesting possibility: websites with their own built-in AI features, powered by your hardware.
I'd love to see this in documentation sites and research tools.
4
5
207
Repository: github.com/amurgola/LumaBrow…
Latest release (v2.2.1): github.com/amurgola/LumaBrow…
Prompt API feature (v2.2.0): github.com/amurgola/LumaBrow…
Local model and hardware docs: github.com/amurgola/LumaBrow…
Agents and MCP documentation: github.com/amurgola/LumaBrow…
3
48









