Prev founding team of: @NousResearch (2023) and @TTSLabsAI (2020) DM for interesting conversations.
S4
Joined March 2021
- Tweets498
- Following841
- Followers6K
- Likes942
We're having a renaissance of American open models, and they seem to be competing not just against Chinese models, but also against closed source labs in the efficiency metrics that matter most: training-efficiency, cost-per-task, speed-per-task, flops-per-task.
Just in the past 2 weeks:
- October 5th: Reflection announced their first model which they're making open: Beam 501B-A23B, trained on 10K+ GB300s, Artificial Analysis has early access and says "Early indicators suggest Beam will be one of the most token-efficient open models we've seen for its level of intelligence." Reflection is also claiming to beat the pretraining efficiency of Deepseek V4 Flash.
Now in just the past 48 hours:
- October 14th: Paper Instruments announces Ultramarine Flash 320B-A18B, an open model RL'd from GLM-5.3-flash that seems to trade blows with the full GLM-5.3 itself, claiming to reach a significantly greater pareto frontier of cost/task in real-world enterprise CUA workloads.
- October 15th, just hours ago today: Nvidia announces a native 2-bit 74B-A9B model, competing with the latest models from china and other labs that have a similar native flops and VRAM footprint.
- October 15th in the past 1 hour: Meta announces Muse Spark Open 816B-A42B, having SOTA score amongst all open models in JobBench, HLE-Diamond, AA-omniscience and MCP-Atlas, while also having video input, audio input, image input and text input. If the original Muse Spark 1.3 benchmarks are anything to go by, this will be very competitive in reasoning efficiency too. (while GLM-5.3 is still beating its absolute scores in most benchmarks, the reasoning efficiency curves might make this SOTA for cost per task and speed per task at similar capability.)
Six years ago in 2020, the same year OpenAI published their seminal neural scaling laws paper, @ajeya_cotra had discussions with Dario Amodei and Paul Christiano about leveraging biological anchors to better forecast AI capabilities, which led to her creating a bio anchor based analysis for forecasting the arrival of TAI (Transformative Artificial Intelligence).
The below image leverages my 2024-2026+ calculations estimating effective compute values and trends from Epoch ECI data (relative to 2020), combined with Ajeya's 2020 weighted mixture visualization for the compute equivalence of evolution, genome, and human lifetime bio anchors.
If the 2024-2026 ECI trend holds, the remaining anchor probability density peaks will be met around:
- Genome anchor ~late 2027
- Medium horizon neural network anchor ~mid 2028
- Long horizon neural network anchor ~2029
- Evolution anchor ~2029
notes:
- Both me and Ajeya were/are using effective compute relative to 2020 for all the mentioned aspects, which takes into account training efficiency gains too, not just raw compute alone.
- I use Epochs observed trend of~15 ECI points increase per year happening since 2024, to calculate the annual effective compute jump for April 2024 and beyond.
-I use my calculations from comparing various recipe-controlled scenarios against ECI, to conclude that a 10 point increase in ECI corresponds to a ~10X increase in effective compute (convenient coincidence but also independently derived by an anon acquaintance)
- I use the GPT-3 model released in 2020 for the effective compute reference, and use Leopolds and EpochAIs in-common math of GPT-4 being ~1,000X the effective compute of GPT-3, to fill in the effective compute gap between GPT-3 and the starting point of the earliest models we have ECI data for, like GPT-4.
For all the capability tracking folks and benchmark enjoyers, we're eating good these past couple of days.
Some of my favorite benchmark/index launches from the past 36 hours are summarized by me below:
DigBench.
A text-only benchmark measuring AI abilities to discover unknown rules in text-based environments, 7 tiers of difficulty. Tier 5 is still less saturated than arc-agi 3 when allowing harnesses in both:
digbench.ai/#leaderboard
Vals Index 2.0.
It is GDP weighted and tracks both speed per task and cost per task: vals.ai/benchmarks/vals_inde…
Vals RSI (recursive self-improvement) index.
This is personally the best attempt I've seen for actually measuring AI abilities to do AI research itself. It tests models ability to do post-training, harness engineering and even how well they can do in the parameter golf challenge, amongst other things:
vals.ai/benchmarks/rsi_index
Anthropic conceptual reasoning index.
Measuring abilities to do conceptual reasoning across multiple different categories like: evaluating the strength of conceptual arguments, having logical consistency in what probabilities the model gives for various things being true, and how well models reason about decision theory. conceptualreasoning.ai/
Epoch EOIS. 400+ open math problems put into a benchmark and tested across each major model. (already over 100 of the problems now solved autonomously by the models as of launch): xitter.cf/tmkadamcz/status/20875…
New benchmark!
I'm releasing OEIS Open, a benchmark of 492 unsolved math conjectures, formalized in Lean.
Given simple tools and $50 per conjecture, Opus 4.8 can already produce Lean proofs resolving 30% of them.
In November 2023, Yann LeCun, Thomas Wolf and others from Meta and Huggingface created a benchmark called GAIA, which described itself as: "A benchmark for General AI Assistants that, if solved, would represent a milestone in AI research." Most of the problem solutions were kept private, not released online.
It proposed 466 "real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency."
On the hardest level, the average human score was 87%, while the leading systems scored less than 3%. 10 months later OpenAI released O1-preview, reaching ~30% on that level.
Now in 2026 the human baseline for the hardest level has officially been surpassed, the best agent systems are now scoring 88.9% on GAIAs hardest level (level 3).
The current highest level 3 score was achieved by Nvidia, leveraging a multi-agent system that includes Nvidias own tool orchestrator model. It scores 89.8% on Lvl 3 (even higher than the 88.9% typo I wrote above)
The public leaderboard can be seen here: huggingface.co/spaces/gaia-b…
This is the best AI-generated short film esque media I’ve seen thus far. Not just visually, but actual storytelling to go with it too.
A common type of joke/argument people will make against this: "My 2 year old is triple the weight compared to when he was born, therefore he will be 2,000 pounds at the age of 32!"
There are two fundamental differences I can think of that make this argument different from FutureEval (this benchmark) and METR time-horizons:
- It's fitting a trend to just 2 data points, as opposed to the dozens of data points that exist for METR time-horizons, and FutureEval.
- It's extrapolating several multiples beyond the span of the existing observational period (taking 2 years of baby growth to extrapolate out way beyond just an extra 2 years into the future), as opposed to METR time-horizons which has ~6 years of observational data and is used to make conversation-worthy predictions, which are often significantly less than 6 years of extrapolation into the future, similarly FutureEval here is extrapolating out into the future by an amount that is ~equal and less than its existing observational span.
This probably depends on the distribution of questions you're testing on, but this benchmark seems quite diverse and atleast difficult enough to the point of the highest scoring models only giving correct answers to ~50% of its 6,000 questions across 42 categories.
A specialized scaffold for creating educational visualizations, I’m a bit surprised it’s able to even do this well with just scaffolding. The natural next step is for someone to train a dedicated model for this, and native audio plus sound fx too. It only gets better from here.
This seems like one of the first unifications of the concept of spiking neural networks, persistent states, attention and adaptive compute, all in one model/“machine”. Great work from Sakana, I believe they’re one of the most under-rated openly publishing labs right now.
Introducing Continuous Thought Machines
New Blog: sakana.ai/ctm/
Modern AI is powerful, but it’s still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence.
Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. We’re exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations.
Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity.
Learn more about our approach:
Interactive Report: pub.sakana.ai/ctm/
Full Paper: arxiv.org/abs/2505.05522
GitHub : github.com/SakanaAI/continuo…
I think quarrels specifically on LLM consciousness are largely a red herring and often don’t end up with any conclusive answer that convinces either side.
The more interesting and practical discussion imo is; Whether it’s ten years, hundreds of years, or millennia from now, it seems inevitable (barring cataclysms) that humanity will eventually create something new that meets virtually any and every criterion of what we’d otherwise call advanced alien life. Along the way we’ll likely have to decide; What criterion do we deem as most important? What to do if there is a blurry spectrum along the way towards that criterion of being definitively advanced alien life or not? If/when/how we should integrate this alien life into our society, how to best prepare for continued future interactions of novel alien life? What differences (if any) we should treat such alien life when it’s from human origin versus non-human origin?
Doing such philosophizing and investigating into these questions now may prove useful for steering the future we desire.