Infra and Incidents @OpenAI. Previously @OrbitalSidekick, @AWS. I build stuff in the cloud and have strong opinions about it.
San Francisco, CA
Joined February 2021
- Tweets590
- Following95
- Followers1K
- Likes212
I'm inclined to agree with this take. Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
Benchmarks don't use skills, memories, and worse often use custom harnesses. They don't make any effort to actually be good at the task and are purely a measure of "can a model one shot this task?" Which is so removed from the reality of working with models today that I can't help but ignore them.
META’S HEAD OF AI SLAMS AI BENCHMARKS
"it’s some PhD student who made these benchmarks"
"you’re kind of outsourcing all of your product thinking to this viral benchmark"
"so clearly that’s wrong"
I think there is something to this. the whole industry developed so rapidly that there was hardly any time to establish proper benchmarking. companies used what was available, competitors then had to catch up on the same benchmarks, and some of them eventually became industry standards
apparently there is quite a lot of randomness and chance in how we are measuring AI progress today
Damn, sounds like they nailed it.
The European Commission has rolled out an alternative to Microsoft Teams.
But officials using it aren't impressed, with one describing the tool as “absolute shit.”
vist.ly/5kje9
Hurry, someone set up a trillion dollar prize for curing cancer otherwise the economics won't be worth it apparently.
Andrew Guenther retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
I have never once had a hotel harass me over a review. I am once again reminded that @Airbnb is a hellscape now.
"Your review is unfair, the overflowing septic tank that made the house smell like shit was simply out of our control"
Get the fuck outta here.
Google is starting to roll out Gemini to Google Home devices and I don't remember the last time I was this angry at a product. How the hell is it so much worse than Google Assistant?
Can't believe ChatGPT Sites hasn't picked up more yet. I can just meme on someone with a prompt. It reminds me of YTMND back in the day.
Andrew Guenther retweeted
i would rather be an optimist and work hard than a pessimist posting about why things won't work.
it's much more difficult and the most likely path is failure, but society fails if people don't try.
no amount of "it will never work" essays will drive society forward.
"In this paper we propose a new benchmark that uses one year old models, poorly configured settings, and a dogshit custom harness to validate our feelings that AI is bad."
Thank fucking god I never went into academia.
It bakes biscuits. I feel like it's pretty self-explanatory.
biscuitsec.org/ 😉
I've been heavily using Voice in Codex to coordinate tasks on remotes the last few weeks and it's quickly become my primary workflow.
ChatGPT Voice is now in the desktop app.
Control your computer and direct multiple agents running in ChatGPT Work or Codex, using just your voice.
It's powered by GPT-Live, so it can speak, listen, and coordinate work in the app at the same time.
Rolling out globally today on macOS and Windows to Plus, Pro, Business, Edu, and Enterprise plans.
