The Dust paper is really big. A breakthrough.
After spending a few days to really digest it though, I'm somewhat disheartened that so much of it is implementation details which are not explained. I'm working on a blog post to reverse engineer the math, methods, and choices.
Backprop has been the only credit assignment algorithm capable of training large neural nets.
Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)
- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.
- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.
- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.
- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.
The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.
w/ @bishmdl76, @cs_serdar, @akshayvegesna
I'm going to be presenting my research at the Latent Space paper club tomorrow, thanks @yikesawjeez.
Link: luma.com/k8u3voz2
As for my origin story, circa 2019 I did my own research on perceptual image hashing methods as an honors college project and ended up discovering deep learning, because it happened to beat every known heuristic. I wrote some tensorflow and pretrained tiny image embedding models on my school laptop. This annoyed my mom, because I left my laptop on the porch overnight so it wouldn't overheat, with a cable running out the back door. I was too poor to afford a GPU at the time, but ended up training some very good models using what little compute I had in the house.
Then I graduated, got a job writing airplane firmware to be close to a girl, then did compilers for a while. I was always doing deep learning projects in the background, but I eventually circled back to my true passion.
I would have still been interested in deep learning without the girl involved, but I'd like to think she helped by dragging me around, inducting me into her friend group, getting me over my stage fright, helping me figure out gender stuff, and forcibly socializing me. I owe her a lot. I definitely would not be the person I am today without her.
I think there are perks to not working at a frontier lab.
You get to think "Oh, well, at those labs they've really got their shit together." They probably do X, and Y, and Z, which are the things which would obviously be done by any Serious Individual working on this problem."
Then you meet people from frontier labs and they tell you "No we aren't doing anything like that at all, that sounds like a lot of work."
Very frustrating, but there is alpha in there somewhere.
I'm finally doing a whole ton of ZO runs, trying to figure out what hparams matter. I've been having success with having agents write mlsweep files, submit them, interpret results, and use the results to inform new experiments. Can save a lot of effort on open ended problems.
>OpenAI awarded us $6,500.
If it was me I would be offended.
I've been thinking about this paper for a couple days.
Neural networks have to do a whole bunch of error correction. They have to. By construction. This is because you're training by feeding in data which is mostly noise from an information theoretic perspective. Computation cannot create signal, only destroy it. The trick is to destroy only the noise, and preserve the signal relevant to the task. Any network which does not maximize this objective would have higher loss. So training must produce models with this quality.
This is not to say, though, that other views on optimization are not correct. I still subscribe to Ilya's idea that pretraining is compression. Compression, too, can be seen as throwing out only the noise and retaining the relevant signal to the task, which can be solved by approximate memorization. But these small changes in perspective are important and useful, I think.
So, from this perspective I got to thinking. Gradient updates must preserve this behavior. What does this imply about the loss geometry? Is there some sort of invariant that can be asserted to build a better optimizer?
I think that there is. I'm still thinking about it. But I feel like this is useful for something.
The smoothed objective from Evolution Strategies and the Taylor-expanded objective from SPSA are literally the same objective, just interpreted through different lenses. Both lenses just happen to produce exactly the same equation.
apaz retweeted
i am in no way a doomer/safetycel/etc. but at a certain point we have to ask where we draw the line with a major lab intentionally hiding the felonies its models have committed from the public.
apaz retweeted
I am interested in optimizers that constrains both the weight matrix's and its update's Frobenius norms. Hyperball is an optimizer that constrains this. As far as I know (not much) though, I haven't seen a full-order exact solution to the problem. Rather, they either provide a first-order solution (in Jianlin Su's blog) or a non-exact solution (Hyperball).
Take Hyperball as the example. The operation is very simple: we first normalize the gradient to have constant F norm, update the weight matrix with the normalized gradient, and then normalize the weight matrix to have constant F norm. However, during the second normalization, the weight update itself no longer has constant F norm. Obviously, we can understand the two normalizations as a shift and a scale, so it is principled and there is nothing wrong with doing so.
However, if we want, we can give an exact solution that guarantees both the weight update and the weight to have constant F norm. It is just a little linear algebra. I provide my derivation as below. The result checks out to be decomposing the weight update to have a weight component and a component whose Frobenius product with the weight is 0.
It is an empirical question whether this formula works on its own, or as a wrapper to existing optimizers like Hyperball. it is another question whether it has already been done before. I don't know but just find this pretty interesting.
Derivation in the blog: riddlehe.github.io/blog/full…
I cannot believe they actually published into the allegations. We need to see receipts. I don't trust anything OpenAI says, and neither should you. I don't know anything about Levent, so I can't calibrate on him. Regardless, we need evidence. I trust nothing else at this point.