Infra and Incidents @OpenAI. Previously @OrbitalSidekick, @AWS. I build stuff in the cloud and have strong opinions about it.
San Francisco, CA
Joined February 2021
- Tweets590
- Following95
- Followers1K
- Likes212
I'm inclined to agree with this take. Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
Benchmarks don't use skills, memories, and worse often use custom harnesses. They don't make any effort to actually be good at the task and are purely a measure of "can a model one shot this task?" Which is so removed from the reality of working with models today that I can't help but ignore them.
META’S HEAD OF AI SLAMS AI BENCHMARKS
"it’s some PhD student who made these benchmarks"
"you’re kind of outsourcing all of your product thinking to this viral benchmark"
"so clearly that’s wrong"
I think there is something to this. the whole industry developed so rapidly that there was hardly any time to establish proper benchmarking. companies used what was available, competitors then had to catch up on the same benchmarks, and some of them eventually became industry standards
apparently there is quite a lot of randomness and chance in how we are measuring AI progress today
Maybe it's my search bubble but it was the first link for "OpenAI API pricing" 😅 Also the OpenAI developers plugin in Codex helps find this info too!
Dots work with cloud tasks and get their own computer. It's taking the experience/way of working with Codex many people have enjoyed/found success in and making it a first class feature that's more accessible. I'm not too surprised personally that the vibe from the Pro crowd is "this is nothing new." It's not, but it also is.
Damn, sounds like they nailed it.
The European Commission has rolled out an alternative to Microsoft Teams.
But officials using it aren't impressed, with one describing the tool as “absolute shit.”
vist.ly/5kje9
I've always used one orchestrator thread to delegate, so to some degree it's just preference. I like dots because they make that orchestrator more powerful. Long term I see dots becoming a push notification sink. Agents are very pull based right now which is super wasteful. Having my dot notified of emails, PR comments, etc, without burning a bunch of tokens on an automation is going to be a big jump. And in that world, it makes sense to have one place for those notifications to go that can delegate for you.
I get more use out of my work dot than my personal. I can tag my work dot in Slack or mention it in a meeting. They really shine as a coordinator. I have less use for that on my personal dot so that experience doesn't *feel* more powerful than a thread on a remote, but the cloud computer and cloud tasks take a step towards making that experience more accessible. I'm not terribly surprised that the reception from the pro crowd is "so what?" since for them it's more a UX difference than a functional one.
And also don't trust niche industry benchmarks because they are often incredibly poorly designed. Internal evals are truly the only way. The gotcha here is that how models respond to prompting varies. We recently rewrote some of our agent prompts from scratch for this reason.
Yeah, it all depends how long horizon your tasks are and how many turns you use a thread for. Play around with it and see what works for your usage. I just go crazy when people talk about 1M context like "big number better"
1M context is good if you're confident you'll never need compaction. The reason it's set lower by default is to give context space for compaction itself. The more space you leave, the better it is, but you compact more often. The defaults are optimal for long running tasks.
Just have your models reinvent all software on every run and be incapable of accessing tools or doing research during evals, that's all it takes!
I'll just leave this here.
status.openai.com/incidents/…
