AI technical writer and poet. Not simultaneously. Research, warm takes, and cool stuff we're doing at @Scale_AI. For poems: @MatthewSiegel_
Joined December 2024
- Tweets92
- Following83
- Followers102
- Likes31
Pinned Tweet
thoroughly excited to launch this video series where I break down new research from @Scale_AI.
this first one is how we knock out 90% of errors in data that require human review using an autorater. check it out 👀
Matthew Siegel retweeted
Interested in collaboration or using Scale’s Physical AI data engine? Contact us at physical-ai@scale.com.
Thanks @LargerLanguage for the help with the blog on Scale Labs!
Big things happening. SEAL is now Scale Labs. Congrats, @scale_AI and @ScaleAILabs teams!
Welcome to the home of all things @scale_AI research — focused on data, evaluation, safety, and post-training that moves frontier models forward.
We’ll share benchmarks, insights, and work intended to be useful to the broader research community.
labs.scale.com?utm_source=hu…
Matthew Siegel retweeted
People talk about “AI Risk” like it’s one giant shoggoth with a single scary face.
It’s not.
It’s six different monsters, each with its own speed, intent, and way of doing damage:
• Adversaries
• Unforced Errors
• Misaligned Goals
• Dependencies
• Societal Impact
• Emergent Behavior
If we treat them like one blob, we’ll overreact to everything or prepare for nothing.
I break them down — clearly — in my latest @scale_AI post with @LargerLanguage
scale.com/blog/shoggoth-ai-r…
Lots of people are talking about an AI bubble. Sure. I don't think these people are reading the same research I'm reading.
Matthew Siegel retweeted
New @Scale_AI paper!
The culprit behind reward hacking? We trace it to misspecification in high-reward tail.
Our fix: rubric-based rewards to tell “excellent” responses apart from “great.”
The result: Less hacking, stronger post-training! arxiv.org/pdf/2509.21500
If you want to get a sense of where AI is going, pay attention to what students are doing. Most of them, it turns out, are using AI for to help them learn.
But are the models they’re using ready to replace human tutors? It turns out, not really.
AI tutors are clearly not about to replace human tutors, and they have a lot of homework to do.
But they also aren’t going anywhere.
TutorBench helps provide a path to help LLMs better support student learning.
Read more about it out on the Scale Blog! scale.com/blog/tutorbench
This was a HUGE lift from our research team, huge thanks to everyone who contributed to this benchmark.
...and the blog doesn't look so bad either:
scale.com/blog/swe-bench-pro
Is your team using LLMs as judges to evaluate AI agents? You might want to take another look at that. When @Scale_AI researchers ran the same evaluation suite on back-to-back days, they saw metrics swing wildly.
