@AnthropicAI. Prev. @Google Brain/DeepMind, founding team @OpenAI. Computer scientist; inventor of the VAE, Adam optimizer, and other methods. ML PhD.
Joined March 2009
- Tweets707
- Following408
- Followers55.7K
- Likes3K
Pinned Tweet
Personal news: I'm joining @AnthropicAI! 😄 Anthropic's approach to AI development resonates significantly with my own beliefs; looking forward to contributing to Anthropic's mission of developing powerful AI systems responsibly. Can't wait to work with their talented team, including a number of great ex-colleagues from OpenAI and Google, and tackle the challenges ahead!
Obviously I strongly support Anthropic's position here.
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement…
1X seems to have the right approach to developing safe humanoid home robots. Developing full autonomy will require lots of in-distribution demonstration data, so this launch, mostly tele-operated, makes a lot of sense. I expect such robots to be ubiquitous in 5-10 years.
I really liked this interview, where Geoff manages to explain the essential bits for a general audience.
I did a podcast with Jon Stewart who has always been a hero of mine. It was a lot of fun. He really wanted to understand how AI works.
youtube.com/watch?v=jrK3PsD3…
Think, people. Like many other technologies, AI technology has applications that are a net negative, unless well regulated. Don't support the anti-regulation lobby and don't support the people and the companies behind it.
Mark Zuckerberg has a vision for how A.I. could be used in Meta's universe.
But the actor and filmmaker Joseph Gordon-Levitt is here to point out a flaw in the technology: an apparent lack of guardrails around how the company's chatbot interacts with underage users. nyti.ms/3ILwCNo
Its kind of personal since
(1) I'm a dad and am concerned for the kids,
(2) I have ex-colleagues who have created a $100M+ blanket anti-regulation superPAC, and
(3) Some of my work is general-purpose and now also used for this kind of bad stuff.
It's already the case that people's free will gets hijacked by screens for hours a day, with lots of negative consequences. AI video can make this worse, since it's directly optimizable.
AI video has positive uses, but most of it will be fast food for the mind.
Very impressed with Veo 3 and all the things people are finding on r/aivideo etc. Makes a big difference qualitatively when you add audio.
There are a few macro aspects to video generation that may not be fully appreciated:
1. Video is the highest bandwidth input to brain. Not just for entertainment but also for work/learning - think diagrams, charts, animations, etc.
2. Video is the most easy/fun. The average person doesn't like reading/writing, it's very effortful. Anyone can (and wants to) engage with video.
3. The barrier to creating videos is -> 0.
4. For the first time, video is directly optimizable.
I have to emphasize/explain the gravity of (4) a bit more. Until now, video has been all about indexing, ranking and serving a finite set of candidates that are (expensively) created by humans. If you are TikTok and you want to keep the attention of a person, the name of the game is to get creators to make videos, and then figure out which video to serve to which person. Collectively, the system of "human creators learning what people like and then ranking algorithms learning how to best show a video to a person" is a very, very poor optimizer. Ok, people are already addicted to TikTok so clearly it's pretty decent, but it's imo nowhere near what is possible in principle.
The videos coming from Veo 3 and friends are the output of a neural network. This is a differentiable process. So you can now take arbitrary objectives, and crush them with gradient descent. I expect that this optimizer will turn out to be significantly, significantly more powerful than what we've seen so far. Even just the iterative, discrete process of optimizing prompts alone via both humans or AIs (and leaving parameters unchanged) may be a strong enough optimizer. So now we can take e.g. engagement (or pupil dilations or etc.) and optimize generated videos directly against that. Or we take ad click conversion and directly optimize against that.
Why index a finite set of videos when you can generate them infinitely and optimize them directly.
I think video has the potential to be an incredible surface for AI -> human communication, future AI GUIs etc. Think about how much easier it is to grok something from a really great diagram or an animation instead of a wall of text. And an incredible medium for human creativity. But this native, high bandwidth medium is also becoming directly optimizable. Imo, TikTok is nothing compared to what is possible. And I'm not so sure that we will like what "optimal" looks like.
Thank you! See you guys in Singapore next week 🥳
👇 Great work led by Yushun (@ericzhang0410) introducing Adam-mini, a version of Adam that, surprisingly, reduces Adam's memory requirement by 50% (!), without negatively affecting convergence rates. Please read Yushun's thread for details!
Finally finished Adam-mini! A "mini" version of Adam that painlessly frees 50% of memory over Adam. Some highlighted features:
1. Adam-mini saves 50% memory over Adam for all modern neural nets. This is done by removing 99.9% Adam's v (but the last 0.1% of v is essential and necessary).
2. Adam-mini's loss curve closely resembles those of Adam. You can almost see no difference!
3. Adam-mini uses the same hyperparams as Adam (lr, beta1, beta2, eps, etc). No tuning is needed!
4. With the free memory you can enlarge batch size and achieve higher throughput (see paper).
5. Principled design: The design of Adam-mini follows a very simple and general principle related to the Hessian structure of neural nets. Such design stems from classical optimization theory (see paper).
Paper: arxiv.org/abs/2406.16793
Code: github.com/zyushun/Adam-mini
Durk Kingma retweeted
In arxiv.org/abs/2303.00848, @dpkingma and @RuiqiGao had suggested that noise augmentation could be used to make other likelihood-based models optimise perceptually weighted losses, like diffusion models do. So cool to see this working well in practice!
Have you ever wondered how to train an autoregressive generative transformer on text and raw pixels, without a pretrained visual tokenizer (e.g. VQ-VAE)?
We have been pondering this during summer and developed a new model: JetFormer 🌊🤖
arxiv.org/abs/2411.19722
A thread 👇
1/
Great blogpost by Ruiqi (and other GDM ex-colleagues), clearly explaining the the connection between flow matching and diffusion models. Super happy they took the time to explain this topic, there's confusion on this topic, I think many will find this quite valuable!
Congrats to @geoffreyhinton for getting the Nobel! His impact is immeasurable, very much deserved.
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”
The recording of our talk for the ICLR'24 test-of-time award (with @wellingmax) is now available online:
iclr.cc/virtual/2024/test-of…
Biggest live audience I've ever spoken to, with >2000 attendees 😅. But it was a lot of fun!