Came away from this amazed at how much latent ability models possess. DeepSeek-V4-Pro solves 51% of SWE-bench Pro with one attempt and 71% with 64 attempts. Efficient inference turns parallel sampling into a lever worth pulling, especially on tasks you can verify.
1/ We think parallel test-time scaling is one of the most underexplored ways to improve model quality
We spend a lot of time on efficiency for long-horizon agentic inference
We showed that a swarm of DeepSeek-V4-Pro agents lifts the SWE-Bench-Pro score by ~20 points, beating the cost-performance trade-off of the leading frontier models🧵
With long-horizon agents, very little of what they send is new (~2% on SWE-bench Pro), so serving them efficiently means holding on to their history instead of reprocessing it.
Full story on running 64 agents through all 731 problems in under a day: blog.doubleword.ai/swe-bench…
While writing a minimal TP setup, I noticed the loss was dropping sluggishly and the gradient norms were unstable vs the single-GPU baseline. Turns out the way I parallelised CE loss ended up erasing gradients.
New post on what went wrong + how to fix it.
Inspired by @jacobrintamaki's posts, I've sketched out some ideas about what could help maximise the project's impact.
@eladgil @shreyanj98
Made another post after having a few conversations with the great @shreyanj98 on evals for this project. It made me believe that to have great evals (if you're paranoid) you need to go all the way back to how you source your PDFs. link below
The recent Kimi paper gives really nice intuition on why MCTS failed for R1.
In AlphaGo the value function was necessary to steer the system towards a strategy that was difficult to exploit—optimal for adversarial play.
For language models straight-line drives of optimal actions seem actually counterproductive. ‘Mistakes’ allow for reflection and backtracking, increasing inference compute by extending the CoT and triggering more forward passes.
Was watching this podcast of @nitin_gadkari with @ANI. I'm comfortable with colloquial Hindi but words like "aarthik" and "svadhinata" had me constantly pausing the video and using Google Translate.
Decided to build an extension to caption YT vids in real time using whisper.cpp.
Current setup - leveraging whisper.cpp server-side for transcription to avoid Whisper API costs. The extension then auto-injects the captions in real time into the YouTube player.
github.com/rushilbhat/YTSubt…