pai_here retweeted
Should you waste tokens having your coding agent use grep, or hand the search to a tiny local model?
We release ColGrep-Agent-2B! A code-search agent that runs locally, searches with ColGrep, reads files, & returns locations + code to your main agent
-> 34% fewer API tokens
pai_here retweeted
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do.
SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡
Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon.
Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.
pai_here retweeted
Intuition: Compaction is agentic recurrence (RNNs), whereas recursion (RLMs) is agentic attention.
Recurrence maintains a constant-size state at all times by compressing the past.
Attention keeps all context fully represented and then decides how to re-process it for each step.
pai_here retweeted
Let Sol manage an efficient fleet of Luna agents for you.
These models know each other well and collaborate to achieve the result in an incredibly fast and efficient way.
pai_here retweeted
One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness. Sometimes I declare it up top, something like "switching to speech recognition sorry for any typos...". Sometimes I turn it into a small interview of a few turns. But I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with. The result is that you improve the mind meld and have to correct things less from that point on.
pai_here retweeted
We benchmarked coding agents on our own internal tasks at Databricks and learned a lot!
There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now. 🧵
pai_here retweeted
Obvious in retrospect, but I didn't really anticipate:
Fable: Performing Final Review of <Awesome Feature>
Also Fable: I appear to have introduced a critical security vulnerability.
<This model's safeguards flagged this message.>
Opus 4.8: Doesn't look like anything to me.
pai_here retweeted
I have been playing around with quantisation for ColBERT (without centroids, we want to get rid of those) for a few weeks and I can confirm all of these results! If you keep the query in a good enough precision, you can compress the documents like crazy. It also works nicely with other compression method such as hierarchical pooling
I think you can even push it further with ternary and delta-encoding. Since we are using hot and cold storage, you can get a « reranking » with a slightly bigger representation stored in the cold bucket (and you can even use the hot representation as original information enhancing your cold one)
It’s so nice to see @mixedbreadai sharing these information, they are leading the productisation of late interaction so they really pushed it! The scoring trick is very nice, I don’t not even think of it, just hamming
I think the biggest bottleneck to push compression even further is to solve the issue of the scores being very close for top documents
The anisotropism makes all cosine pretty similar and summing them converges towards low gap in scores. We can compress like crazy without changing the scores much, but the gaps are so small it hurts ranking…
We are so early!
pai_here retweeted
There's a whole RLM ecosystem out there. Some noteworthy ones:
• @a1zhang's rlm — the paper's reference implementation
github.com/alexzhang13/rlm
• @DSPyOSS dspy.RLM — my go-to for benchmarking RLM
github.com/stanfordnlp/dspy
• @dosco's ax — first-class RLM in TS, very "agent native" approach
github.com/ax-llm/ax
• @irl_danB's unix-rlm — filesystem as the sandbox / REPL - pure bash
github.com/openprose/unix-rl…
• @OpenProseVM's prose — declarative markdown contracts with RLM-inspired Reactor harness
github.com/openprose/prose
pai_here retweeted
Same in France, it's a nightmare.
I have a translation guy I email and he goes to his village city hall and says to the mayor that his translation is legit so it is OFFICIAL. I have to pay €50 for the privilege. It confuses everyone outside europe.
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
pai_here retweeted
Brilliant idea! Next up: Apple randomly reboots your Mac if you're building competing tech, Gmail silently edits your email if you mention rival platforms, and Tesla Autopilot swerves if it detects you're working on self-driving cars.
All in the name of safety, of course. Because malicious actors controlling the world’s operating systems, inboxes and cars would be extremely dangerous!
pai_here retweeted
These are some of my LLM assisted contributions from the past month. Nothing amazing, but I'm slowly getting better at it. Atm, using Qwen3.6 27B exclusively. For hardware - switching between M2 Ultra and RTX 5090. Both are good options, though after using the RTX and going back to the Mac, it always feels like a snail. Yet for most tasks, I feel like both hardware can do the job comfortably.
pai_here retweeted
NEW: malware developers added nuclear & biological weapons text to to their spyware.
Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner.
Cleanest practical example I can think of for why over-indexing on first order safety alignment is risky.
When closed (and open) models ship with aggressive refusals, they will be sprinkled with second-order blindspots that attackers will discover...and exploit.
We are only in the earliest days of attackers leveraging these features, and it wouldn't surprise me if users systems that need to handle complex cybersecurity issues demand that models be less safety-blunted.
In the weeds: @SocketSecurity's post also shows why intention matters in how you design a malware analysis pipeline to avoid prompt manipulation.
H/T to colleagues that shared this with me socket.dev/blog/mini-shai-hu…
pai_here retweeted
such underrated content I felt I had to mention it again, idk any of the authors personally but @tedzadouri explains and motivates FA4 in an extremely intuitive way
afaik this is now the most beginner-friendly (?) piece of content going over FA4 right now
Great technical talk by @tedzadouri on FlashAttention-4: a deep look at how attention kernels are being redesigned for NVIDIA Blackwell, where the bottleneck shifts from tensor cores to softmax + memory movement.
Also featured: voice of god aka @marksaroufim in the background asking questions.
Link below!
Behind the MiMo API Price Reduction:
The deepest price cut, up to 99%, is for Input (Cache Hit). The core reason is our inference framework now supports hierarchical KV cache optimization for SWA. Production inference engine tests show this optimization increases cached token capacity by 5x, equivalent to an 80% reduction in caching costs. Combined with Cache Read Overlap among multiple Full Attention modules in the Hybrid model, actual costs are further reduced.
Prices for Input (Cache Miss) and Output are also reduced by 60%-80%. This mainly benefits from the extreme 1:7 Full:SWA sparsity ratio brought by the model architecture (the prefill compute of the 70-layer MiMo-V2.5-Pro roughly equals a 10-layer GQA model). This kept our original inference costs well below the industry average, naturally leaving a 2x-3x profit margin in pricing. This price adjustment simply reflects our decision to pass these structural cost efficiencies directly to developers.
Operating at these newly reduced API prices, our production inference engine is running at near full capacity, and we can still essentially break even. We previously advised LLM companies not to "blindly cut prices" precisely because very few model architectures and inference optimizations can keep API costs from running at a loss. If more architectures that save compute and KV cache emerge, along with better inference Infra to drive down API costs, this will form an excellent virtuous cycle in the industry.
More crucially, affordable, high-performance model APIs will drive real, sustained, and at-scale inference demand. This upstream demand pulls forward the development of the entire AI infrastructure chain—including chips, servers, optical transceivers, PCBs, liquid cooling, power, energy storage, and data centers—serving as a strategic fulcrum for a systemic revaluation of AI hardware. In the long run, this injects more affordable and accessible compute into both training and inference pipelines, accelerating the parallel evolution of global AGI across multiple regions and technical routes.
For more technical details, we will release a detailed Blog post later.
pai_here retweeted
For anyone who isn't sure, this is how you release a model and talk about the performance. Not 3-5 cherry-picked benchmarks.
Replying to @Alibaba_Qwen
Performance:Qwen3.7-Max performs strongly across benchmarks in coding agents , and improves massively in general-purpose agents. Qwen3.7-Max also demonstrates exceptional strength on the hardest reasoning benchmarks, and stands out in general capabilities and multilingualism.
pai_here retweeted
Over at Substack, @JoshEakle asks: "It's 2026, and I have yet to see an anti-almond farm protest."
Claude Code v2.1.36+ quietly added an x-anthropic-billing-header block at the start of the system prompt.
It includes a random 5-character cch value.
If you are using Claude Code with Ollama, LM Studio, or a third-party proxy, that tiny changing field can break prompt cache hits.
Same prompt.
Fresh cache miss.
Full prompt reprocessed.
Slower inference.
Fix:
Add this to ~/.claude/settings.json 👇
pai_here retweeted
The only correct answer when a VC asks: "What's your moat?"
Starting a company in a garage is boring so we started @dottxtai in a French castle instead








































