@WecoAI

The platform for self-improving code

weco.ai
Joined April 2023
  • Tweets94
  • Following7
  • Followers2.8K
39Media
I'm a bit shocked that our RSI system (AIDE²) actually discovered a similar mechanism. I initially thought it was a weird mechanism that had slipped through due to some lucky evaluation noise. But after reading this more rigorous study, it now makes sense to me.
Dimitris Papailiopoulos
@DimitrisPapailOct 12
Fork-and-Flush: help your agents escape Idea Basins!! I love thinking of AutoResearch as Idea Descent: a metric of interest goes down, as a function of ideas/experiments, where the agents get pulled towards one idea or another. But we've seen very often an agent develop this weird tunnel vision, and honing in just one type of "idea", kinda like GD stuck in a local minimum. We studied this and seems we can fix it with Fork-and-Flush, a super simple and kinda cute idea. Every so often, fork the agent into T copies in the same workspace but empty the context, let them run for a bit, and continue from the best one. If you like the GD analogy, it's basically basin hopping en.wikipedia.org/wiki/Basin-…. BTW we found a silly way of also plotting "idea basins": an LLM rewrites every experiment as a spec and answers a few thousand binary queries "is Experiment A closer to B or to C?". Then we embed this NXN matrix into 2D and indeed see in many cases that each run saturates into a small region and keeps polishing it. We call these idea basins! At equal compute Fork and Flush beats single runs and best-of-N on 12/13 long-horizon tasks (some 4+ days), and in my own research projects it's been working really well! BTW we only use it at test time, but it could also be part of training or RL. Led by @jackcai1206 @christos_ziakas during Jack's internships with us, along with @vkontonis @ShivamGarg91462 @Tea_Pearce Sid Sen and Akshay Krishnamurthy.
11
21
2
306
12,721
Conversation between our cofounder&CEO @zhengyaojiang and @ecsquendor on: - Living with the alien spaghetti code discovered by autoresearch - Why we’re not yet in the singularity yet. - Our research on reward hacking (SpecBench) and how RSI cut the reward hacking rate
Machine Learning Street Talk
@MLStreetTalkOct 12
> An AI rewrote its own coding agent for 8 days. > @zhengyaojiang post calling it the first evidence of recursive self-improvement got 1.8M views here. And no, he doesn't expect a singularity!
8
545
If an AI research agent improves its own harness using one model, does the improvement still hold on other models? The arXiv paper on AIDE², our RSI system, tests this and compares it with more AI research agents.
Zhengyao Jiang
@zhengyaojiangSep 23
We're releasing the arXiv paper on AIDE². The RSI system where AI research agents improve their own research efficiency. It includes new results on transfer across models and comparisons with more AI research agents: [1/4]
2
18
1,250
Can autoresearch agents find better training data, not just better training code? Here’s our new EMNLP paper: AutoData: Agentic Search for Pre-training Data Selection [1/6]
7
14
5
122
19,169
more replies
Paper has been up on arxiv and code are opensourced: Paper: arxiv.org/abs/2609.19754 GitHub: github.com/WecoAI/AutoData Blog: yanmeng-nlp.com/blog/auto-re… [5/6]

AutoData: Agentic Search for Pre-training Data Selection

LLM agents have recently shown promise in automating machine learning engineering by editing model and training code under execution feedback. Data, however, remains largely outside this agentic...

arxiv.org
1
1
13
585
Work led by @vivian_yanmy during her internship at Weco. We'd also like to thanks to @NVIDIA for the compute support! [6/6]
8
482
I think the AI/ML researcher's role will be greatly transformed by autoresearch, but it won't disappear. It boils down to three skills: - Coming up with creative primitives - Defining a good abstraction for the agent to search within - Defining a good eval on what good is With the RSI results we published this week, this becomes even more relevant. My @aiDotEngineer talk makes the full case, dedicated recording now up: youtu.be/iCj_ATyThvc

How Autoresearch is changing ML research — Zhengyao Jiang, Weco

Earlier this year, OpenAI ran Parameter Golf, a model-training comp...

youtube.com
2
11
2
96
18,075
Awesome work by @WecoAI in the pursuit of RSI. Proof the small companies innovating with taste and determination can move fast, and move well!
Zhengyao Jiang
@zhengyaojiangJul 14
The first experimental evidence of recursive self-improvement (RSI). Autoresearching the autoresearch agent for eight days. The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)
1
7
56
10,944
Exciting agent harness recursive self-improvement result. Congrats @zhengyaojiang, @yuxiang and @WecoAI team!
Zhengyao Jiang
@zhengyaojiangJul 14
The first experimental evidence of recursive self-improvement (RSI). Autoresearching the autoresearch agent for eight days. The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)
4
21
193
33,529
Weco AI retweeted
George
@odysseus0zJul 14
RSI of harness by harness for harness! Already producing a harness that is great at doing auto research! Phenomenal work and progress.
Zhengyao Jiang
@zhengyaojiangJul 14
The first experimental evidence of recursive self-improvement (RSI). Autoresearching the autoresearch agent for eight days. The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)
2
3
26
4,271
Thousands of search trees. 99 rewrites of the agent's own code. 9 in 10 rejected by a held-out gate. Eight unattended days. The AIDE² pilot report is out. AIDE_85, the discovered agent and more detailed PDF tech report, will be released when the remaining analysis lands
Zhengyao Jiang
@zhengyaojiangJul 14
The first experimental evidence of recursive self-improvement (RSI). Autoresearching the autoresearch agent for eight days. The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)
1
5
26
4,353
Sharing our framework for thinking about RSI, with some experimental results coming soon!
Zhengyao Jiang
@zhengyaojiangJul 10
Everyone assumes RSI would lead to an intelligence explosion. Is that really the case? We tried to clean up the logic after 8 months of building RSI at the harness layer. Not all RSI is equal. We split it into 4 levels: delegation, net positive, ignition, and inflection. (1/9)
1
7
1,941
We ran autoresearch on preprocessing for a fraud-detection task. With a loose API the score looked great, and the code was leaking test data into the features in 13 different ways. We tightened the abstraction to a strict API and the reward-hacking rate dropped to zero. My metaphor @aiDotEngineer talk: doing autoresearch is like training a model. The codebase abstraction is the architecture, it biases everything the search can find. The eval is the loss function and the data. Human's research & engineer's role is no longer write exact code but get the constraints and feedback signal right. Link to the talk in reply
3
10
1
69
7,212
Big fan of @WecoAI's work on automated discovery💡 We worked with AIDE, when extending it for the AI Scientist-v2 🧑‍🔬 and the OAI parameter golf results for the collaborative Aiden agent look really impressive 🚀 📝: weco.ai/blog/parameter-golf-… Highly recommend checking out @zhengyaojiang's talk at @aiDotEngineer 👀 👉 youtube.com/live/4sX_He5c4sI…
1
14
1
47
3,555
Weco AI retweeted
Abby@anmorgan2414Jun 25
7/ "Hands-on AutoResearch: Cracking OpenAI's Parameter Golf" (@WecoAI). Train the best LM in 16MB (int8 + zlib), 10 min on 8xH100s, scored on bits-per-byte. Weco's autoresearch agent outperformed 1,000+ participants. Workshop has you run the agent yourself.
1
1
6
426
Weco Autoresearch is now steerable, and we allow Claude Code to connect to the dashboard
Zhengyao Jiang
@zhengyaojiangJun 25
One of our autoresearch runs sat flat for 60 steps. One message got it moving again. Everyone is pushing research agents toward full autonomy, but what helps most is being able to step in when a run goes wrong, without breaking the loop. (1/6)
4
896
Load more