The hottest new programming language is English

First step to replacing transformer is appreciating it deeply for how far it was able to carry us

Project Genie is a @GoogleLabs experiment that lets you simulate dynamic worlds you can navigate in real time with Genie, our general-purpose world model. Today, we’re connecting Project Genie to nearly 20 years of Street View data from Google Maps — so you can now build interactive spaces based on real-world locations. Street View imagery in Project Genie is available now for places in the U.S., and will expand to more locales over time. #GoogleIO

This is an extremely beautiful plot because it sheds light on why scaling laws are so smooth, and reconciles empirical findings from both scaling laws and grokking. Even though mean loss across all tasks (red line) decreases smoothly, we see that individual subtask losses drop in a much more phase transition / grokking-like way. Easier subtasks are learned first, harder subtasks learn much later. This is also consistent with many LLM evals where you see a sharp inflection as you scale, rather than smooth linear improvement. It also suggests why robotics models have not yet seen sufficiently convincing scaling laws: even though we may collect enormous numbers of demonstrations & hours of data, the data do not contain enough diversity or "underlying subtask quanta" for the losses to "meld together" to form a clean scaling law. It remains a mystery why natural data, sorted by subtask difficulty, seems to form a Zipfian distribution. Are tasks we perceive as "difficult" actually nothing more than "infrequent" in our training distribution? There is relationship between the length of the shortest program that can generate some data, and the frequency of that data. If a subtask were *more* frequent, then you could actually shorten the program needed to generate it, thereby making the task easier from a Kolmogorov Complexity POV. And if you assume my previous claim about robotic scaling laws is true, why makes robotics data have such bad zipfian coefficients? Does the coefficient only get "good" once you do the tokenizer + dedup + operationalize the data collection just right? Does a hard subtask take more steps to learn because it requires representations from "easier" subtasks to be learned first? Representational dependency enforces a strict ordering in tasks that can be learned (task B cannot be learned until task A is mostly learned, task C cannot learn until B is mostly learned)? This could explain why there is an "ordering" of difficulty - it arises from natural ordering of dependencies in distributed representations like DNNs. It would be interesting to study this in a synthetic context, where one has the power to tune frequency of data independently of "minimum description length" as defined by P(Internet text), and see if it is possible to "learn hard tasks faster". Lots of potential in applying a better mechanized understanding to how scaling laws form to improving scaling properties on frontier models.