the new sonnet3.7 review as someone who has used it before: - the new sonnet is great, on our internal evals on rust we see 14.7% (40% around) improvement (this eval is made up of 1k questions) - it has a stronger affinity to end of context instructions, we add a reminder of the tool format at the end of our prompts and it was over indexing on it a bit which was weird but nothing out of the ordinary - It does a good job at implicit planning: this one really made me happy, I can even throw away o1 right now and just use the new sonnet model - we did see doom loops appear post 150k input tokens but this is still better than the 70k input tokens of the older sonnet - its great at finding its way around issues, even when taking a wrong approach its about to course correct and does not get stuck on local maxima - the terminal usage has clearly improved a LOT! - one nice behavior from sonnet3.7 was that it did not need to read the file again after editing to make a new edit again a bit surprising when I first noticed it but looks like showing it the git-diff was enough for it to understand how to go about making more edits on the same file again (this is big if you are working on agentic code generation) - the visible COT is a big difference, when it comes to debugging. Gone are the days for hoping that the system prompt will do the right thing or where it is going wrong. This made me lean into reasoning models more - 1.0 temperature was the preferred setting, altho I am not sure if that changes with the model release today @AnthropicAI did a stellar job with sonnet3.7 I am glad that I can use this model again, been waiting on this for weeks

We are sunsetting Aide. as an editor we did pretty good, hit some big numbers and were burning through more than 10B tokens just yesterday the sad part is, we got out executed and out gunned by our competition some of my most memorable memories: - being SOTA on swebench twice - learning about the internals of VSCode and syncing those changes upstream when we could - people coming over to our discord surprised by how good the agent is compared to everyone else - fighting smut content on our servers (XD) so what's next? we as a team know how to ship fast (real fast) and iterate quickly, we will be taking a stand against Devin and believe our agent is truly special (pun intended). To everyone who has used Aide and supported us, I want to say thank you. It was a tremendous challenge which Naresh and I took on ourselves and started back in 2023 and kept at it until earlier this year. we will share about our agent_farm and what is in store for the future of AI codegen tools :) I am personally very excited about this space and will share more soon onwards and upwards!

this week at codestory: - DDOS attack on our servers - smut content generated round the clock - agent got wings, works for longer and is more stable - <=90% of code on sidecar was written by the agent itself :V

if you asked me a month ago.. I would say agentic llms would be good until 20 steps a month later and I am putting in checks for >= 200 steps (agentic + reasoning models) still tinkering with the setup but good for a full eval run wild times indeed

👑

We have seen 3 new thinking models to date: - o1 - r1 - gemini-2.0-flash-thinking-exp-1219 What really excites me here is that unlike the static cost of running training, we are now moving the cost margin to inference PS: we are still not solving counting r's in 🍓 (yet!)

If you are still on the fence about using Aide

youtu.be/u-g0sT6wm5E?si=_QFp… dropping more alpha on building an editor and how we reached SOTA on swebench-lite