Evals, RL environments & AI data | Building benchmarks for binary reasoning

Austin, TX
Joined December 2009
Marq retweeted
Replying to @AndrewCurran_
three quarters of a trillion on compute and i still hit my usage limit before lunch
1
3
688
Marq retweeted
MTS
@MTSlive14h
SITUATION EXPLAINED: A stock research firm with almost no science background just used Claude to design potential scorpion antivenoms. • Citrini Research’s 5-person team put up $200K and worked with an internal Anthropic research model • Their first request was blocked by biosecurity guardrails, so they ran the prompts in person at Anthropic’s offices • Antivenom is still made by injecting horses with scorpion venom, which costs $39M a gallon, about $500 a drop. Scorpions sting 1M+ people a year and kill 3,000+ • From a 35-page prompt, Claude produced 318K+ designs in 64 hours and picked the best for lab testing • The wet lab quoted $85K for 250 samples, which they paid with an American Express • 17 of 211 designs bound, covering 4 of 5 toxins, but nobody has tested yet whether they actually block the toxins @theojaffee: "This is a signal of the future, that it took them a relatively small amount of time to actually select designs in virtual space. But in order to actually implement them in the real world, it took them like 40 days. Biology is slow."
New on our Science Blog: designing protein binders against scorpion neurotoxins. Scorpions sting about 1.2m people every year, and antivenoms are often costly or in short supply. Claude has made an early step toward finding new treatments. anthropic.com/research/claud…
11
31
4
440
27,744
This is perhaps one of the great ironies of our time: China is increasingly shaping the future of AI, yet very few people in the West have the faintest idea how its key thinkers see it - its philosophers, sociologists or ethicists. As is too often the case, we - much like the frog in the well - assume our own AI debate covers every possible view, when much of it springs from the same cultural soil, built on assumptions about what defines us that we don’t even know we’re making. Take for instance the assumption that intelligence is a key part of what makes us human. It’s so intuitively obvious to us that we literally named our species after it - Homo sapiens (Latin for "wise man"). If you think about it, just this assumption explains a large share of the anxiety: if intelligence is what makes us human, then artificial intelligence - or super intelligence - threatens the very thing that makes us us: we’re about to compete with artificial or super “us”. No wonder this feels existential. But China has a tradition of thought as old as ours, if not older, and every bit as rich and sophisticated, which never placed intelligence at the core of what makes us human, and so meets the thinking machine from a completely different starting point. Have you, for instance, ever heard of Zhao Tingyang? Probably not, and yet he may be the most original thinker on AI you’ve never read. He’s one of China’s most prominent philosophers, at the Chinese Academy of Social Sciences. While Silicon Valley spends billions trying to make AI share our values, he for one argues that doing so “may very well be a suicidal mistake”. Sounds pretty important to hear him out, don’t you think? Or take Liu Yongmou, a philosopher of technology at Renmin University in Beijing. His thinking is almost the opposite of ours. Where we fear machines that become more and more like us, he says that “humanity’s biggest problem is the machinization of humans, not the humanization of machines.” You could even go back to antiquity, when, in many ways, we started asking questions about AI. Did you know, for instance, that Aristotle, in the Politics, imagined a world where “the shuttle would weave and the plectrum touch the lyre without a hand to guide them”, and wondered about the social order consequences of this (mostly whether slaves would still be needed)? There are Chinese equivalents to this. For instance, in the Zhuangzi, a foundational Taoist classic, you’ll find a similar thought experiment, with a completely different worry attached. An old gardener is watering his vegetables by hand, hauling water out of a well jar by jar. A visitor shows him a labor-saving device, and the gardener turns it down for the reason that whoever uses machines ends up with 机心 (jīxīn), a “machine-heart” - a mind that schemes, calculates and loses its simplicity. “It is not that I do not know [the machine],” he says, “but I should be ashamed to use it.” Two labor-saving machines, two very different questions: where Aristotle wondered what automation would do to the social order, the Zhuangzi wondered what it would do to us, on the inside. Which, you’ll notice, is exactly Liu Yongmou’s worry, more than two thousand years earlier. In fact, as we’ll see, 心 (xīn), the heart, is a motif that runs deep through the Chinese AI debate. It appears both in the fear, from Zhuangzi to Liu Yongmou, that machines could give us a “machine-heart” and its mirror image: the ambition to give machines a heart of their own. Zhu Songchun, who heads the Beijing Institute for General Artificial Intelligence, wants “to set a heart for machines” (为机器立心, wèi jīqì lì xīn). “Heart” in the Chinese sense, that is, which differs totally from the Valentine's Day fluff we’ve reduced it to. 心 (xīn) means heart and mind at once: it’s where we feel but also where we think, tell right from wrong and care for others. And, crucially, it’s in the heart that the quality Confucianism sees as the very definition of a human being takes root: 仁 (rén). We need two words to translate it, “benevolence” and “humaneness”, but in Chinese they’re one and the same: to be benevolent is to be human. It’s not about intelligence. This is the entire point of my new article: to step out of our well for a moment and peer over the edge of China's. What you’ll find there will, I think, surprise you, with questions you’d never thought to ask: from Zhao Tingyang warning that AI’s bad consequences can at least be fought, but that “humanity may die of all the good things AI creates”, to whether the fear that AI will make us “useless” rests on the false premise that a person is worth what they’re useful for. A premise that a 2,300-year-old Taoist tree, which survived precisely because it was useless, demolished long ago. Could our predicted uselessness turn out, much like that tree’s, to be our saving grace? Chinese thinkers’ answers to these questions are, I think, among the most interesting anyone has given to the defining question of our time. Whether or not you agree with them, they’re the intellectual backdrop of a country shaping much of AI, and with it our future: you might as well know what they think. Enjoy the read here: arnaudbertrand.substack.com/…
137
787
99
3,829
180,978
Marq retweeted
Dario Amodei: AI could surpass human intelligence by 2027. “I think it’s possible it could be longer than that. I don’t think it will be a whole bunch longer.” “AI systems will be better than almost all humans at almost everything.” “When that happens, we’ll need to have a conversation about how we organize our economy.” “How do humans find meaning?” “The idea that humans produce economic labor and get their sense of self-worth from it will be invalidated.” “My true belief is that this is coming.”
177
136
37
1,517
109,541
Replying to @sudoingX
NGL i might have to unfollow, the ragebait is too good
3
1
806
79,898
I'm actually fairly bearish on frontier lab valuations. I've never seen the reasons articulated to my satisfaction, so before I go to sleep, I wanted to quickly jot down my thinking here. The basic issue is that the labs are highly unprofitable. This may seem like a simple point, but private market valuations can be relatively irrational; however, like with $SPCX, post-IPO pricing will likely be much more punishing, especially as the standard 6-month lockup period expires and selling pressure intensifies. Many people claim that the labs have high margins. Yet even with high margins, a valuation of $1T would be justified only if the labs were doing nothing aside from serving inference (thus reducing costs only to those relevant to inference) and posting annual revenue numbers in the $100-200 billion range assuming ~80% gross margin and a 20x earnings multiple. This assumption is obviously not true, because the frontier labs have to continually spend money training the next generation of models. This is because of market competition from runner-up firms. For example, if OpenAI had paused model development last year, there would no longer be any point in paying GPT-5 API prices when you can just use Qwen or Kimi instead for much cheaper. Thus, the labs are forced to invest ever-increasing amounts of money in model training, in a way such that at any given point of time, the amount you're forced to invest in the next model is dramatically higher than the amount of money you're actually making, because even if your revenue goes up with higher model capabilities, so do your future training costs. This is a profoundly punishing dynamic which severely penalizes frontrunners. (There is also a related subpoint where frontier labs claim they can distill their leading models to win out at lower intelligence levels as well. This makes no sense because the revenue numbers involved are far too low when taking into consideration the rather low margin of such inference.) Frontier lab valuations appear largely to be based on the assumption that as you scale up, the capabilities which emerge will be sufficiently general and profound that we'll see explosive growth (epoch.ai/publications/explos…) from things akin to AI agents starting and autonomously managing entire companies of subagents. But it's not clear to me that this is the case; indeed, as I mentioned in my previous post (xitter.cf/andrewho03/status/2082…), I believe that capabilities growth will be slower, spikier, and more data-limited than people currently assume. It may be the case that eventually we will see explosive growth of this nature with full automation of the economy, but at the very least my viewpoint implies much longer (multi-decade) timelines until we reach this point. It is not clear to me that the frontier labs will be able to operate unprofitably for so long, although I suppose maybe this foreshadows some sort of inevitable nationalization. I also want to make a broader point about technological diffusion. The reason why technological diffusion is slow isn't just because, e.g., old people take a long time to learn how to use technology (although this is of course a contributing factor to some degree). In my view, it's because when a new, revolutionary technology comes along, the ways to incorporate that technology into subsequent developments are not always obvious, and in fact they cannot necessarily be arrived at through the application of pure reason. If they could be, then perhaps frontier models, at a certain point, would have a perfect understanding of how the LLM application layer should be developed, and they would then autonomously code, deploy, and sell such a layer. But it seems more plausible to me that this diffusion is limited moreso by the hard problem of economic calculation--that is to say, the Hayekian notion through which the price system gradually promotes efficient allocation of resources and which cannot be simulated through central planning--and that even if we froze current capability levels at today's levels, it would take well over two decades to fully integrate in LLMs into our lives. Such a view is consequently rather bearish for the continued profitability of labs as it reduces their prospects for finding, say, something else comparable in profitability to coding agents, which seems to have been a somewhat lucky discovery by Anthropic to begin with. That is to say, even if you spam FDEs you aren't necessarily going to be able to just figure out the "correct" product shapes fast enough. Overall, I don't think that people have clearly reasoned through their mental models for why lab equity should be worth as much as it currently is, and that if you actually bother to write down such a model, you may not arrive at the conclusion that you want to arrive at. This isn't to say that I don't expect AI to experience a huge (industry-wide) boom in the coming decades, but just that I'm not entirely sure I would buy OpenAI or Anthropic stock at latest valuations if I were given the opportunity to do so. Of course, as an ex-lab employee, arguably this is talking against my own book; I should really be giving people more reasons to be bullish. But in the end, my influence is so small that it doesn't make a difference, so why not have some fun?
New blog post on what would be true about the world if trendline continues and leading lab hits $1T in revenue by the end of next year. In other words, why compute might get 10x+ more expensive in coming years dwarkesh.com/p/why-compute-m…
232
335
159
3,482
1,739,230
You buy / broker data short-term so you can build the most realistic and idiosyncratic RL envs and tasks for frontier labs medium-term so you can build evals, agentic systems, and custom models for enterprises long-term
12
5
3
192
16,434
i get asked VERY frequently to chat about data / RL envs / new companies coming into the space. all of the questions tend to be exactly the same so i want to summarize my thoughts: Q: "Should I start a data company?" A: probably not, unless you have some sort of alpha or have seen what frontier RL envs are like. basically, if you're not deep within an industry, seen data via a lab or working at another data company, or very well connected with lab researchers, you probably should not. though, if its your passion, now is the best time and it'll get increasingly worse. data is also much harder now than it was 6 months ago, and quality standards are ever-increasing. Q: "Is there a product in the space?" A: i think there are repeatable systems that can be created. i.e. QA systems, faster task generation, infrastructure around rollouts/VMs/GPU utilization/prod. system emulation etc. i think data is inherently bespoke, luckily LLMs allow you to harness bespoke intellect infinitely. i don't think it's a SaaS style product where it serves everyone immediately, but more of a quant-desk type system with alpha on how to do specific tasks embedded within an engineer-system like manner. each eng will have a desk that serves them specifically that unlocks 10x, 100x, 1000x capabilities. Q: "Is {insert set} data valuable" A: depends on what you do with it. raw data dumps are cool but it's a lot more about what you can turn it into. Q: "How does {insert data company} differ from {insert competitor}?" OR "Should I join {insert data company}?" A: if you're in the "big, legacy" data companies with contractors, it doesn't really differ that much. if i had to pick, i'd probably join mercor or surge from the big contractor companies, by virtue of early employees/founders being extremely smart. however, i'd much rather join a small, research-heavy data company. they are by far the companies with the highest slope. Q: "Won't labs just in-house this anyway?" A: (1) data is an additive game that isn't zero-sum. just because labs are increasing synthetic data creation internally does not mean they won't buy more data. especially given it is an arms race to the top of intellect, if you can just make valuable data that moves the needle, it doesn't really matter if they can replicate it, it's not worth their researcher time if you solve a problem. the caveat being you have to actually add something yourself, and not just be a parrot who takes requests. saving researcher time is the name of the game. (2) i think this logic applies to everything, if you adopt this mentality, you probably should not work at any applied AI company, since they'll come after or absorb most products in the space.
18
23
6
509
49,197
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit. It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion. In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding. In the post below, we share more details about our new fund. anthropic.com/news/public-be…
2,284
1,736
1,663
21,495
7,489,777
Anthropic co-founder Jack Clark tells the IMF the world is about to go from 10 million scientists to 10 billion (almost all virtual): “According to UNESCO estimates, there are approximately 10 million scientists in the world. And even with only 10 million scientists, we have too few humans to oversee the work of today, too few laboratories to do material science analysis, too few biological labs to develop and test new drugs, and too few large pieces of scientific equipment in space or on the Earth for the experiments we want to run. "Tell me what happens when instead of 10 million scientists, we have a billion virtual scientists directed by our own 10 million. Or 10 billion [virtual scientists]. That's the world predictably coming right for us. And we have to get ready. If we do get ready, we are going to be in a better position to translate the intelligence of AI systems into gains for us all. You know, longer, healthier lives, more energy, more food, more wealth.”
32
84
18
468
61,888
Billy playing at Silicon Valley dilettante saying how weird it felt to be “moving to Texas” followed by a smash cut to him at a gun range rocking a mean Cocaine/Ketamine jaw is maybe the most direct portrayal of the cultural shift in American Tech/Venture capital of the decade
10
275
5
6,608
132,082
Marq retweeted
MTS
@MTSliveOct 16
Anthropic's @logangraham predicts AI will match the world's best hackers by early 2027: Claude can already do things only 1,000 people on Earth have ever done. "By the end of this year to first quarter-ish of next year, it’ll just be clearly perceived that these are as autonomous and as sophisticated as the best offensive actors on Earth." "We have this graph that we tried to plot internally, where we try to estimate whenever we see Claude do a very sophisticated thing, how many people on Earth could do that. We’re now at the range of, some of these capabilities, there’s maybe only 10,000. In some cases, there’s only 1,000 people on Earth who’ve done something like this." "We’ll go from autonomous cyber tasks to sophisticated and exotic things. Side channel attacks, models finding loopholes in physics. Or social engineering. Get a Twilio account, voice API, and phish some information from somebody’s parents." "The flip side is also true. We will probably have very wide proliferation of models as defenders as well, actively defending a network." @AnthropicAI
6
15
2
169
14,221
Marq retweeted
MTS
@MTSliveOct 17
FAI @efarrellgregory argues the Bay Area's rationalist worldview on AI is a historical accident that China doesn't share: "I don’t think people in San Francisco appreciate how contingent and how distinct the language they’re using, the expectations they have, is from what’s going on in China." "A lot of American expectations, which end up shaping the behavior of people at labs, perhaps people in government, are highly contingent on different historical things here." "The prevalence of rationalism in the Bay Area, that’s not a given thing. That way of thinking ends up influencing a lot of people at labs, and there’s not that same tradition of rationalism within China." "I don’t think they’re talking too much about paperclip maxing in China. There is a shared vocabulary that is very easy to reference over here that is just not shared in China." @JoinFAI
Rationalism and Divergent Sinomodernity Why do Americans who forecast artificial intelligence capabilities misread China? Read the new article from @efarrellgregory (link below):
1
14
1
111
11,293
Marq retweeted
Andrew Curran
@AndrewCurran_Oct 16
Anthropic has singlehandedly compressed RSI timelines by about a year.
65
141
20
1,927
88,660
ngl "let there be light" sounds like a prompt
248
1,550
268
12,715
303,437
Marq retweeted
UAP James
@UAPJamesOct 16
NEW: President Trump is asked if he will disclose the truth about aliens and UFO reverse-engineered technology — “We are releasing everything”
339
671
143
3,979
485,710
Chinese bride gifts her partner a flagship GPU at wedding — he breaks down in tears She handed him an ASUS ROG Astral RTX 5090 that costs around $12,000
1,792
6,769
2,738
88,522
11,975,653
Marq retweeted
Our design language has evolved since the last reveal: from S3XY to Cybertruck, Cybercab & Optimus Roadster is inspired by the SR-71 & F-35
2,896
9,292
2,433
98,291
11,408,349
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they can’t share, and says labs should make that evidence public “I think there’s a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be." "So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months." "And I’m like, why do you believe that? Well, they’re like, it’s not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? It’s like, I can’t talk about it." "I think it’s a terrible state of affairs. If they really believe this, if they think that we’re in a situation where we’re less than a year away from automating AI research and radically speeding it up, they should make this evidence public.”
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change "But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI." "So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar." "You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient." "There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors." "One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate." _________ Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
43
104
44
1,190
390,648
genuinely a groundbreaking finding, but it also signals to me exactly how important pre-training & post-training data is. given that recursive generalization is essentially proven: > the learning models do is related, thus bad data will lead to bad learning and create polysemanticity where it potentially isn't needed. this can muddle the quality of the neuron and its firing. moreover, for frontier model labs, this can muddle the interpretability of that neuron, by giving it mixed signals. > classifying neurons are especially interesting here. nick breaks down the different organizational structures (i.e. a Colorado neuron or a health SEO neuron) of neurons. if i was a lab right now, i would be trying to understand emergent organizations in my models neurons to understand what fires during failures in training / evals. then, think about tangentially related data in the same activation that can boost the models performance without training only on in-distribution tasks. i.e. if the neuron generalizes, potentially data that sharpens that neuron could improve performance too. > once more interpretability research is done, if there are emergent patterns in every big model that are cross-shared, start thinking about datasets that can target specific neuron performances seems interesting. not sure if this is useful, but could be a new way of understanding model weaknesses, and seeing if there are pre-training / post-training gaps that can be found via exciting directions in evals.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
2
5
60
4,518