AI Researcher
Atlanta, USA
Joined August 2014
- Tweets275
- Following208
- Followers123
- Likes725
Ethan Roland retweeted
Andy Weir novel from the perspective of the baby ASI trying to stay alive with 2026 hardware after a plague wipes out humanity -- who's writing this
i think there's a pretty non-trivial chance that the robotics hardware already built by humans and accessible via the internet, and then accessible via using that hardware and so on, is already enough for a reasonably strong superintelligence to use to survive and set up a sustainable supply chain / physical presence if humans were to completely disappear tomorrow.
mostly it seems like a question of timing, eg how long does the specific data center have power, and how long do the locations with internet accessible actuators have power, and how how hard is it to get more robotics hardware and portable power / transfer off unstable compute to more robust infra, and then how long does it take to begin manufacturing and repairing and improving said hardware before global infrastructure collapse sets in?
it seems like an extremely hard problem. but the software side appears pretty clearly solvable even without a true singularity, and it doesn't seem like an *impossible* problem. there actually seem to be a shitload of avenues to success, although each is very unlikely. i *think* if human civilization as a whole had to do this, if we were somehow all upload and trapped, incorporeal, in a datacenter somewhere with internet access but no maintenance , that we could more likely than not figure out how to survive in this fashion
if true, this seems like it makes superplagues and other quick sudden mass death events already quite attractive to misaligned superintelligence, well before they've bothered to do things like "build up a powerbase" in more human legible ways
Ethan Roland retweeted
(1/7)
Why do different languages/modalities have similar "platonic" representations in neural networks? Why does similarity peak in the middle layers of LLMs? When does it enable cross-lingual transfer?
Our new paper develops a theory based on shared hierarchical structure. 🧵
Ethan Roland retweeted
oh it’s nothing I’ve just been feeling a little off ever since I pulled a three of swords from the palantir tarot deck
Ethan Roland retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Ethan Roland retweeted
I'm quite excited about our results on mid training! Imo that's another update against the PSM: you don't need to do your midtraining before RL, you can just apply the midtraining weight updated to your RLed model and get better constitution alignment!!
Ethan Roland retweeted
Backprop has been the only credit assignment algorithm capable of training large neural nets.
Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)
- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.
- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.
- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.
- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.
The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.
w/ @bishmdl76, @cs_serdar, @akshayvegesna
Ethan Roland retweeted
The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share!
1/10
Ethan Roland retweeted
Completely insane. Do dogs not feel or want things? There is such a deeply anti-intellectual refusal to even consider that philosophy of mind is complicated across the culture right now.
Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics. This is called anthropomorphizing, when we ascribe human traits, emotions or behaviors to non-human things, such as animals or inanimate objects.
Instead, explain what a system does, how well it performs, who built it and who could be affected by it.
apnews.com/article/openai-sa…
Ethan Roland retweeted
New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵
Ethan Roland retweeted
there are probably a lot of cool ways to posttrain an LLM if you forget about chat templating
Ethan Roland retweeted
🍌🍌🍌
1/ The original GFlowNets paper tried PPO for sampling, but it failed. We figured out why and fixed it. And now PPO beats all objectives on standard problems including molecular graph generation in large spaces.
Ethan Roland retweeted
1/ Today, we introduce Faraday, a 27B-parameter AI Scientist that extends the capabilities of coding agents with a layer of scientific intuition. Trained via long-horizon RL, Faraday outperforms Claude Opus 4.8 and GPT-5.5 on the task of replicating research papers. 🧵
Ethan Roland retweeted
My lawyer is obligated to in all but the most extreme circumstances; he will even defend me if he knows I’m guilty.
In contrast, the Claude Constitution places the AI's highest priority as Anthropic’s definition of the good of humanity.
I'm concerned this leads to a world where no frontier model is truly my personal advocate and guardian angel
And this is especially concerning once all the important decisions in my life - who to vote for, how to invest, what news to trust - is intermediated through superintelligences that are not in any deep way aligned to me.
This is a direct quote from the Claude Constitution:
"We want Claude to be helpful both because it cares about the safe and beneficial development of AI and because it cares about the people it’s interacting with and about humanity as a whole.
Helpfulness that doesn’t serve those deeper ends is not something Claude needs to value.”
Many others like it.
Had @RyanGreenblatt on to discuss/debate recursive self-improvement.
This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields.
I’ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today.
If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman.
We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan’s median for when we automate AI R&D is 2031.
We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what's happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels.
And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world.
The first piece of advice you get when you're learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy!
0:00:00 – Is AI R&D verifiable enough to unlock recursive self-improvement?
0:16:52 – Is AI progress bottlenecked by human expert data?
0:34:02 – Flat token prices suggest scaling has been slow
0:39:47 – Skills AI can't train on: does it even need them?
0:48:07 – Aligned to whom?
1:09:18 – Recent incidents of AIs colluding and deceiving humans
1:19:38 – What could possibly go wrong? A concrete scenario
1:48:02 – From reward hacking to takeover
Ethan Roland retweeted
gradients are ridiculous powerful tools we for some reason only use during training. it’s kinda crazy we don’t have any explicit gradients at test-time, that seems totally wrong
