Thrilled to share our latest work in @Nature studying inter-brain neural dynamics in both biological and artificial intelligence systems — amazing work by @xzh_brain, Nguyen Phi, and fantastic collaboration with @JonathanCKao! Read the full article here: rdcu.be/euu5Z

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

We have to take the LLMs to school. When you open any textbook, you'll see three major types of information: 1. Background information / exposition. The meat of the textbook that explains concepts. As you attend over it, your brain is training on that data. This is equivalent to pretraining, where the model is reading the internet and accumulating background knowledge. 2. Worked problems with solutions. These are concrete examples of how an expert solves problems. They are demonstrations to be imitated. This is equivalent to supervised finetuning, where the model is finetuning on "ideal responses" for an Assistant, written by humans. 3. Practice problems. These are prompts to the student, usually without the solution, but always with the final answer. There are usually many, many of these at the end of each chapter. They are prompting the student to learn by trial & error - they have to try a bunch of stuff to get to the right answer. This is equivalent to reinforcement learning. We've subjected LLMs to a ton of 1 and 2, but 3 is a nascent, emerging frontier. When we're creating datasets for LLMs, it's no different from writing textbooks for them, with these 3 types of data. They have to read, and they have to practice.

An eventful day, but I am not personally worried for the following reasons: The new DeepSeek models show a number of advances that make training more efficient, and I expect those to be quickly incorporated by the leading US labs, giving them some additional performance benefits. There is a vast amount of innovation out there that makes model training more efficient, including some of our own, like GaLore, that my lab developed together with Meta, but it has not yet been incorporated in any public Llama release. While there are a large number of advances that are similar to these out there in open research, incorporating them into the biggest model runs is a risky endeavor, since every single one comes at a risk of breaking the training run if it doesn't actually work. The DeepSeek paper omits to detail the cost of all those failed attempts and only accounts for the final successful one. For this reason, it is very tempting for some of the bigger labs to just scale what they know to work bigger without incorporating any risky new approaches. If that continues too long there is a chance that one misses out on a number of gains that compound which has happened here. I expect in the future that in addition to the big training runs (which I fully expect to continue), labs will now invest in more mid-size training runs on top of that to try out more experimental techniques to not let this happen again and benefit from open research more quickly. In terms of impact on the chip market, this work further reinforces the approach of test-time computing for reasoning, which will require significant resources. Since DeepSeek showed further innovation and modifications to model architecture will continue to happen, it will remain advantageous for general-purpose GPUs as opposed to narrow specialized inference chips. While DeepSeek is already available on many GPU-based hosting providers, they so far seem to be absent from the ones with specialized chips, despite the fact that the DeepSeek base model has been available for almost a month. This shows the importance of the entire GPU stack to quickly adopt new innovations.

Article: newsletter.languagemodels.co…