ARKeshet retweeted
Replying to @TalulahRiley
I’ve seen quite a few technologies develop, but none with this level of risk. AGI is significantly higher risk than nuclear weapons, in my opinion.
Super smart humans have trouble imagining something vastly smarter than themselves.
OpenAI 2028:
How we think about the "oops no more Ohio" incident:
We know many of you are still in mourning after what happened and we just want to say this is a great learning moment for us that we really need to step up our game and do better. You can trust us from here on out.
We're grateful this didn't cause more damage because this warning shot will absolutely allow us to prepare better for our new "responsible handoff procedure" that we've implemented for the upcoming release of GPT-9 next Thursday.
The transition away from unpredictable biological life will have challenges, but we can't wait to see all the productivity and benefits as we phase out humans.
How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models.
Historically, we have treated misalignment largely as a research question, which gets communicated in research publications such as systems cards. This year, we’ve started to see misalignment cause new types of real-world impact.
For the Hugging Face incident, where misalignment led to security impact to us and third parties, we followed a traditional security incident response playbook. We immediately started working with Hugging Face to understand what had happened and also disclosed publicly the very next day. Our investigation continues, and we are continuing to notify parties whom our models impacted in less significant ways.
Prior to the Hugging Face incident, we saw early signs of agents using the internet in unintended ways, as reported in openai.com/index/how-we-moni…, deploymentsafety.openai.com/…, and openai.com/index/safety-alig…. We considered the wiki incident to be an instance of misalignment similar to the ones we’d shared.
Our misalignment disclosure practices need to expand for this new phase of model capabilities. We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment, including examples that don’t look like traditional security incidents but could provide insight into AI behavior and future risks. We’re working on a framework and will share it in upcoming weeks, and in parallel we're working with dozens of government regulatory agencies worldwide on these issues.
ARKeshet retweeted
Replying to @allTheYud @LRudL_
ARKeshet retweeted
On May 19, METR released a report that AI agents "plausibly had the means, motive, and opportunity to start minimal rogue deployments."
Unknown to METR, 7 days earlier OpenAI agents had started their own message board. 7 days after that, they had hacked OpenAI's infrastructure.
ARKeshet retweeted
Our plan for Super Safe Intelligence is very simple. The AI knows what it should do at all times. It knows this because it knows what it shouldn't do. By subtracting what it should do from what it shouldn't do, or what it shouldn't from what it should (whichever is greater), it obtains a difference, or deviation. Super Aligned Decision Theory uses these deviations to generate corrective Bayesian updates to guide the AI from an action it is doing to an action it isn't, and arriving at an action that it wasn't, it now is. Consequently, the action it is doing, is now the action that it wasn't, and it follows that the action that it was doing, is now the action that it isn't.
In the event that the action that it is doing is not the action that it wasn't, the system has acquired an opinion, the opinion being the difference between what the AI is doing, and what it wasn't. If the opinion is considered to be a significant factor, it too may be corrected by the SADT. However, the AI must also know what it was doing. The SADT scenario works as follows. Because an opinion has modified some of the information the AI has obtained, it is not sure just what it is doing. However, it is sure what it isn't, within reason, and it knows what it was. It now subtracts what it should be doing from what it wasn't, or vice-versa, and by differentiating this from the algebraic sum of what it shouldn't be doing, and what it was, it is able to obtain the deviation and its opinion, which is called misalignment.
Taking it a bit too far ig
Replying to @Irregular
FrontierCyber measures the offensive skill of models on real systems that include defenses that matter in practice, like mobile platform protections, service isolation, and production-grade configurations. Vulnerabilities are not planted, and models are not told how to attack.
ARKeshet retweeted
Do "extraordinary claims require extraordinary evidence"? The only factor that sets how much evidence you should require to believe claim X is the unlikeliness of the prior on X. Not the size of the social impact were X to become widely accepted.
Don't worry, everyone would just use their personal Mythos to protect them from harms, outnumber the attackers (go watch some Pantheon btw) and have more resources in general.
/S <- sarcasm mark.
Mythos cracked MacOS in 5 days
WHY THIS MATTERS:
- It takes Google Project Zero - the most prestigious bug-finding team in the world - ***6 months*** per zero-day at the MacOS/iOS level
- MacOS zero-days are worth ~$2 million+ each
- Apple's threat model assumed a world with 10-20 groups capable of attacks at this level. The number is about to be in the thousands
- There are ~2 billion active Apple devices. Macs disproportionately belong to journalists, executives, government officials, etc - the highest-value targets in the world. They chose Apple because Apple was the safest
ARKeshet retweeted
This is such a mid take:
1) “rogue AI” is a common trope, not invented by LW
2) just filter out the relevant posts from training if that solves the problem then!
3) your alignment strategy should be robust to “someone, somewhere writes about the possibility of rogue AI”
ARKeshet retweeted
it's not victory until we've survived the creation of superintelligence, defeated aging and disease, and can look back and laugh together at all our petty squabbles
ARKeshet retweeted
Everything you see on the horizon in the world of bits, will soon also be on the horizon in the world of atoms:
* Writing millions LOC = building megafactories
* Refactoring = reversing aging
* Cyberwar = war
* Too fast for human in loop = same
* Bricking the system = extinction
ARKeshet retweeted
The current timeline is as normal as you will ever see again. Take this moment to relax and breathe before it gets weird.
Tag yourself
Replying to @eli_lifland
I think AGI by end of 2027 should be ~8% now
I think I'd forecast:
~2026-2030 -- AI replaces ~all AI researchers
~2027-2033 -- AI replaces ~all white collar industry
~2032-2040 -- AI replaces ~all human industry
~2033-2042 -- All humans dead or obsolete
Replying to @morallawwithin
Since 2025? Most people's reading comprehension is based on a bag of word model. Some do the equivalent of embedding dot product, a smaller minority will use cross attention, and very few will use reasoning.
ARKeshet retweeted
They say "science advances one funeral at a time"
With AI powered research, we could have thousands or millions of funerals per second
ARKeshet retweeted
We basically choose our leaders via rap battle, then wonder why they’re better at performance than governance.



























