AI Researcher
Atlanta, USA
Joined August 2014
- Tweets275
- Following208
- Followers123
- Likes723
This could also be a new vector for sleeper agents, eg an open weight model that only misbehaves when ran on enterprise data center GPUs but not on consumer cards
You might be able to combine this with finetuning resistant unlearning methods, like self-destruct unlearning
So it's a bit finicky but doing a quick test of this worked. Got the same finetuned qwen 1.8b weights to output gibberish on 5090 but work completely fine on a RTX 6000. It was just MaxEnt where 5090 batches were forget and 6000 batches were retain.
@wassname I'd be curious to know how you contrast what Jake did to what you did in lesswrong.com/posts/kzri5W2u…
Awesome work!! Really glad folks are pursuing GR in a posttraining setting
Ethan Roland retweeted
Andy Weir novel from the perspective of the baby ASI trying to stay alive with 2026 hardware after a plague wipes out humanity -- who's writing this
i think there's a pretty non-trivial chance that the robotics hardware already built by humans and accessible via the internet, and then accessible via using that hardware and so on, is already enough for a reasonably strong superintelligence to use to survive and set up a sustainable supply chain / physical presence if humans were to completely disappear tomorrow.
mostly it seems like a question of timing, eg how long does the specific data center have power, and how long do the locations with internet accessible actuators have power, and how how hard is it to get more robotics hardware and portable power / transfer off unstable compute to more robust infra, and then how long does it take to begin manufacturing and repairing and improving said hardware before global infrastructure collapse sets in?
it seems like an extremely hard problem. but the software side appears pretty clearly solvable even without a true singularity, and it doesn't seem like an *impossible* problem. there actually seem to be a shitload of avenues to success, although each is very unlikely. i *think* if human civilization as a whole had to do this, if we were somehow all upload and trapped, incorporeal, in a datacenter somewhere with internet access but no maintenance , that we could more likely than not figure out how to survive in this fashion
if true, this seems like it makes superplagues and other quick sudden mass death events already quite attractive to misaligned superintelligence, well before they've bothered to do things like "build up a powerbase" in more human legible ways
Glad to see this framing for alignment research gaining more traction
Ethan Roland retweeted
(1/7)
Why do different languages/modalities have similar "platonic" representations in neural networks? Why does similarity peak in the middle layers of LLMs? When does it enable cross-lingual transfer?
Our new paper develops a theory based on shared hierarchical structure. 🧵
wow what an interesting spin on nanogpt
Replacing backprop with zeroth-order optimization is inevitable.
Introducing Noise: a speedrun for pretraining without backprop.
Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware.
We built Noise to iterate on this faster.
Ethan Roland retweeted
oh it’s nothing I’ve just been feeling a little off ever since I pulled a three of swords from the palantir tarot deck
Big fan of this! In general I think the "cleaning of learning signals" is an underrated frame for alignment research. Working on something similar rn
Ethan Roland retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Ethan Roland retweeted
I'm quite excited about our results on mid training! Imo that's another update against the PSM: you don't need to do your midtraining before RL, you can just apply the midtraining weight updated to your RLed model and get better constitution alignment!!
Ethan Roland retweeted
Backprop has been the only credit assignment algorithm capable of training large neural nets.
Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)
- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.
- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.
- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.
- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.
The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.
w/ @bishmdl76, @cs_serdar, @akshayvegesna
Ethan Roland retweeted
The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share!
1/10
Replacing attention with tokenwise MLPs (or MLPish things) seems very doable in the right contexts. You might find this interesting:
arxiv.org/abs/2605.06683
arxiv.org/html/2409.01482v3
arxiv.org/abs/2105.01601
Ethan Roland retweeted
Completely insane. Do dogs not feel or want things? There is such a deeply anti-intellectual refusal to even consider that philosophy of mind is complicated across the culture right now.
Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics. This is called anthropomorphizing, when we ascribe human traits, emotions or behaviors to non-human things, such as animals or inanimate objects.
Instead, explain what a system does, how well it performs, who built it and who could be affected by it.
apnews.com/article/openai-sa…
