Co-Founder @boolean_ai // Prev. Head of Fraud at @RobinhoodApp
Palo Alto, CA
Joined March 2020
- Tweets137
- Following59
- Followers146
- Likes6.1K
Congrats to @sahillp , @vineetgoel92 and the rest of the team
Huuuuge success !!!!
Context rot is very real
@sanyamsatia found a very clever way of measuring its impact and along the way also discovered some "catastrophic failures"
;tldr use `/compact` or `/clear` as soon as you finish a task
How much does context rot cost on agentic coding tasks? We prefilled 250k–500k tokens of context and ran GPT-5.6 Sol and Claude Opus 5 on terminal-bench tasks to find out.
Sol performance drops significantly at longer contexts, even when the context is completely unrelated to the task. Opus 5 holds flat on unrelated context but degrades when the context is related.
Sol also experiences catastrophic failures at long context: it gets confused while working on the new task and suddenly answers a question from earlier in the context and ends its turn! 🤯
Full post: boolean.ai/blog/context-rot-…
Omar retweeted
How much does context rot cost on agentic coding tasks? We prefilled 250k–500k tokens of context and ran GPT-5.6 Sol and Claude Opus 5 on terminal-bench tasks to find out.
Sol performance drops significantly at longer contexts, even when the context is completely unrelated to the task. Opus 5 holds flat on unrelated context but degrades when the context is related.
Sol also experiences catastrophic failures at long context: it gets confused while working on the new task and suddenly answers a question from earlier in the context and ends its turn! 🤯
Full post: boolean.ai/blog/context-rot-…
Omar retweeted
This is my experience as well
Just because a model can support 1m context doesn’t mean you should use that much context
I try to limit all the models I use to 200-250k context max
How much does context rot cost on agentic coding tasks? We prefilled 250k–500k tokens of context and ran GPT-5.6 Sol and Claude Opus 5 on terminal-bench tasks to find out.
Sol performance drops significantly at longer contexts, even when the context is completely unrelated to the task. Opus 5 holds flat on unrelated context but degrades when the context is related.
Sol also experiences catastrophic failures at long context: it gets confused while working on the new task and suddenly answers a question from earlier in the context and ends its turn! 🤯
Full post: boolean.ai/blog/context-rot-…
Terrible take
DO NOT come to Palo Alto
I repeat
DO NOT come to Palo Alto
Very scary place
It's time for a South Bay renaissance
Might be hard to recall, but used to be not that long ago that the center of Silicon Valley was squarely in Palo Alto, not SF. When I came to the Bay in 2010, that's where I lived and that's where I stayed and settled down to build my startups
It's a beautiful place to live, maybe the best weather in the world, teeming with top talent, safe and family friendly.... but the locus of tech has moved to SF., leaving the startup scene down here somewhat bare and deprived. Most of us are too busy to make it to SF more than 1-2x a month.
I'm committed to revitalizing the South Bay for startups - with events, dinners, social activities (incl parent/kids activities), etc. We can build a vibrant hub down here if a few of us make it happen.
If you're with me and want to be a part of this renaissance, reply or DM me and let's make it happen
Labs are eating through evals faster than ever
Great piece by @sanyamsatia on how to unlock the "eval bottleneck"
boolean.ai/blog/evals-are-th…
1/ We are saturating benchmarks faster than ever before - the time to saturation for benchmarks has gone from 4+ years to less than 1 year after publication in the last few years.
boolean.ai/blog/evals-are-th…
Great insights from @sanyamsatia on why most evals fail, with a simple recipe on what makes a good eval
"I've found that good evals have the same shape usually: they are easy to understand, have a simple prompt (few lines at most) and are simple to verify, but require a lot of work"
Omar retweeted
FrontierBench is live! Excited to have been a task author and reviewer on this effort, with @boolean_ai as a data partner.
The team was rigorous in keeping the quality bar high at scale, across a wide range of domains. Congrats to @ryan_marten and everyone involved on the launch!
Omar retweeted
We ran a frontend eval from an in-progress internal benchmark. Kimi K3 is not at the same level as current frontier models like GPT 5.6 Sol or Fable 5. It's closest to Opus 4.7 on this eval so it's 3 months behind frontier. An impressive result nonetheless.
Evaluating frontier models is going to increasingly require very high taste and in-depth domain expertise.
Big news: Kimi-K3 by @Kimi_Moonshot is now #1 in the Frontend Code Arena with 1679 pts, surpassing Claude Fable 5.
This is a 17-place jump from Kimi-k2.6 (#18 -> #1).
In Frontend, Kimi-K3 ranked #1 in 6 of 7 domains: Brand & Marketing, Reference-Based Design, Data & Analytics, Consumer Product, Simulations, and Content Creation Tools, landing #2 only in Gaming behind Fable 5.
The full model weights will be released by July 27.
Congrats to the @Kimi_Moonshot team on this major milestone!
A world where the best models are never released to the public and are only used internally by large labs won’t be as fun
