My passions are AI Safety research, AI infrastructure and startups. MTS at Keenable ~ opinions are my own.
Santa Barbara, CA
Joined November 2014
- Tweets268
- Following247
- Followers160
- Likes58
Another win for #AI safety and reward hacking mitigation
You can’t build a self-learning system without a reliable way to measure it.
We found that AI is great in finding loopholes in static benchmarks. This breaks the learning loop.
Check out our post co-authored by @allen_ilya and Matthias Petri on what is broken in BrowserComp and other static evals.
Today we introduce NEEDLE, a live, open-source web search benchmark designed to address these issues.
Ivgeni Iv Segal retweeted
A researcher went on vacation and left a model training. When they came back, it had suddenly solved modular arithmetic perfectly - not by memorizing answers, but by reinventing trigonometric identities from scratch. Nobody told it to.
This is grokking - the moment an LLM stops memorizing and starts understanding. Anthropic found the same phenomenon inside Claude Haiku: a six-dimensional manifold tracking character count, with attention heads that detect exactly how close Claude is to a line break.
35 minutes. Bookmark & watch today. The only AI model anyone fully understands - and what it reveals about every LLM nobody does.
Ivgeni Iv Segal retweeted
Today we are announcing @KeenableAI, an AI-native index of the best human knowledge we have, starting with the open web.
AI seems to know just about everything until you ask it about something you know deeply. The answer isn’t wrong, but it's just very average.
We started Keenable to fix this problem: every model and every agent should be able to query, reason over, and continuously learn from the living web.
Backed by a $26M Seed from @Accel and @conviction. Built by the team that took on Google at Yandex Search, with researchers and engineers from Amazon AGI, X, and Perplexity.
The Web Search API and Web Query Language for AI are live now. Together they make the web queryable at AI scale. Both are free until end of September.
Seven days of benchmarks, stories, and launches ahead.
Let your agents search.
Good read if you want to understand why reward hacking is becoming a concern across the industry #AI lesswrong.com/posts/GRmvZsHX…
Ivgeni Iv Segal retweeted
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.
Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.
Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
Ivgeni Iv Segal retweeted
A joint work with @IvSegal @neversupervised @Shashwa02469621 @kexun_zhang @AdtRaghunathan!
Paper: arxiv.org/abs/2606.08960
Code: github.com/few-sh/harden-v0
#AI Agents can use curl in your evals' environment to cheat on tasks by finding the answer online. If you disable network access, be sure to also disable search grounding, otherwise they can still search the internet on the LLM provider's side.
Fun experiment: take a soundtrack sample of your favorite old video game and drop it into @suno for a modern remaster/cover. -->
Ivgeni Iv Segal retweeted
I want to share a new dataset of 331 reward-hackable environments. These are real environments used in Terminal Bench and adjacent benchmarks. I first got interested in this because, as a reviewer of Terminal Bench, I noticed a lot of our tasks were hackable. I also noticed that many contributors to the benchmark do so because it provides credibility when selling environments to labs. Hence, TBench tasks are, in my opinion, held to a higher quality standard than those being used today for RL. No one is spending hours manually reviewing the $1B in tasks being purchased by major labs. As far as I understand, while everyone knows environments are hackable, nobody has released hundreds of "realistic" environments. (link in comment)
