We think it’s great that lots of people are reviewing and scrutinizing DeepSWE. It being our first benchmark, we wanted to invite public collaboration and discourse, which is why we shared all our tasks and grading results in the open.
Since then, we've made lots of improvements in our methods internally and I'm very excited about the problems we’re better able to solve as a result. Notably, we’re at a point again where no benchmarks accurately reflect the capabilities of current models, and our upcoming work will be addressing this.
been running into some frustrating issues with cf artifacts - anyone that can help? @dillon_mulroy @dok2001
Been seeing some pretty impressive improvements over the last few gemini model versions. Can definitely see a use for this now, especially given the cost and speed.
A very impressive open model
GLM 5.2 is now on DeepSWE as the top open-source model on our leaderboard.
With a pass@1 score of 44% at max effort, GLM 5.2 is indisputable #1 open-source model besting Kimi K2.7 Code by 17%.
Charley Lee retweeted
We've updated the Artificial Analysis Coding Agent Index, replacing SWE-Bench Pro with Datacurve's DeepSWE benchmark - the swap lifts Codex with GPT-5.5 (xhigh) above Claude Code with Opus 4.8 (max), while the newly released Claude Fable 5 (max) in Claude Code debuts at the top
DeepSWE, built by @datacurve, writes its tasks from scratch rather than adapting them from public GitHub issues or pull requests, so no model has seen the solutions during training. That matters because SWE-Bench Pro, the benchmark it replaces in our Coding Agent Index, had grown gameable, with some models recovering the fix from the repository's commit history instead of solving the task.
The swap reorders the index: Codex with GPT-5.5 (xhigh) rises from 65 to 76, overtaking Claude Code with Opus 4.8 (max) at 73. Claude Code with Fable 5 (max), which enters directly on the refreshed index, leads at 77. SWE-Bench Pro had been flattering some combinations and penalizing others.
More below.
Charley Lee retweeted
This is the first code bench that actually aligns with how it feels to use these models coding.
Excited to launch this as a measure of performance on truly uncontaminated SWE tasks
So many coding agent interfaces treat great code review UX as an afterthought (understandable for CLIs, but even GUIs??) when it’s such a critical part of the process
It's been a bit since we started as UncleGPT
Congrats to @datacurve on their $15M Series A!
Datacurve provides frontier training data to the world’s leading foundation-model labs, helping to push the boundaries of what AI can do.
techcrunch.com/2025/10/09/da…
Charley Lee retweeted
Today we’re announcing we’ve raised $17.5 million in funding across a $15M Series A led by Chemistry and a $2.7M Seed to accelerate foundation model progress through providing frontier training data for LLMs.
When we first started Datacurve, it came from a simple realization: foundation model progress is limited not just by compute, but by data quality and complexity.
The right data unlocks new capabilities, especially in coding, where accuracy and reasoning matter most.
We’re now proud to partner with the world’s leading foundation-model labs, providing them with high-quality, complex training data that helps push the boundaries of what AI can do.
This is still just the start. Come build the future of technology with us in San Francisco: datacurve.ai/careers
Huge thanks to our incredible team and investors who’ve believed in us since day one and beyond: @garrytan at @ycombinator, @1vnzh from @cohere , @Mark_Goldberg_ from @chemistry_fund, @TheDerrickLi from @AforeVC, @forwarddeploy, @SoheilK, and @shyamalanadkat.
guy who's only ever done hackathon projects: "webdev is so easy, i'm probably as good as some senior SWEs out there"
finally able to increase my uncle count and get rich with unclegpt 🤑
Yesterday, our team said we’re not gonna sleep until we launch.
Announcing UncleGPT - a middle aged AGI. With UncleGPT, you can now create large-scale projects that are 70% automated by AI. You be the visionary, let AI do the work.
Try it out here: unclegpt.app/
I wanna try this 🤣
I gave GPT-4 a budget of $100 and told it to make as much money as possible.
I'm acting as its human liaison, buying anything it says to.
Do you think it'll be able to make smart investments and build an online business?
Follow along 👀
