0xsd retweeted
man, he is just doing the meme about not wanting solutions and just wanting to be angry
nothing will suffice for this guy, his worldview is dogmatic and unshakeable and no matter what comes of mech interp developments, he will still believe we are all going to die
it's sad
amazing
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
imagine hating the technology that produces this
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
0xsd retweeted
even though "onchain" hasn't meant anything at all for many years...
maybe you shouldn't let things that can have a shut down announcement use that word
cryptographic world computer
The cryptographic world computer:
vitalik.eth.limo/general/202…
My attempt to express in somewhat concise terms the true meaning of basically everything planned to happen to Ethereum starting from the fork after Hegota. It's really not just a blockchain anymore. It's a hybrid architecture that combines together blockchains and modern cryptography, to enable much more powerful properties.
FOCIL, EIP-8288, Lean consensus, state management, formal verification, advanced mempool improvements (including privacy), and the longer-term specter of obfuscation all mentioned.
cool
Looks like today may be a record day for token volume % of open models on Vercel AI Gateway:
🟦 Open 78.4% 🟨 Closed 21.6%
While spend 💲 usually tells a different story, #3 and #4 today are Moonshot AI & DeepSeek. Adding Z.ai, their combined spend surpasses OpenAI (#2).
(Do note that's the spend for inference of the model across providers (mostly in the US), not revenue going directly to the open weight labs.)
0xsd retweeted
>be me
>discover effective altruism
>apparently normal charity is inefficient
>why donate to random sad thing when spreadsheet can tell you optimal sad thing
>fair enough
>buy mosquito nets
>save lives
>numbers look good
>feel powerful
>couple years later
>someone asks an innocent question
>why only count people alive today
>huh
>future people matter too
>obviously
>my grandchildren shouldn't matter less just because they haven't spawned yet
>reasonable.jpg
>keep following logic
>what about their grandchildren
>also yes
>what about people in 500 years
>sure
>5000 years
>why not
>500 million years
>starting to get weird but morality is morality
>open calculator
>humanity could survive for an astronomically long time
>could colonize galaxy
>could have trillions upon trillions of descendants
>maybe digital people too
>maybe simulated civilizations
>maybe dyson spheres full of happy uploaded minds
>calculator starts smoking
>realize currently living humans are rounding error
>8 billion people suddenly looking extremely beta
>future contains potentially 10^something people
>can't even fit beneficiaries in google sheets
>new moral priority unlocked
>protect the long-term future
>stop thinking in units of "people helped"
>start thinking in "fraction of cosmic endowment preserved"
>malaria?
>terrible
>but only kills existing humans
>AI extinction could delete the entire light cone
>nuclear war could permanently derail civilization
>bad institutions could lock in terrible values for ten million years
>someone invents wrong constitution in 2140
>quadrillions suffer
>better fund governance workshop now
>friend says maybe we should improve hospitals
>explain opportunity cost
>friend says hospitals are full of actual sick people
>explain scope sensitivity
>friend stops inviting me to dinner
>need to decide what to fund
>easy
>expected value
>suppose project has one in a million chance of preventing extinction
>sounds tiny
>but extinction destroys 10^50 future lives
>multiply
>mother of god
>$10 million project has expected value of several galaxies
>charity evaluation complete
>someone asks where the one-in-a-million number came from
>expert judgement
>which expert
>us
>how calibrated
>extremely thoughtfully
>reduce estimate to one in ten million to be conservative
>still beats curing cancer by 38 orders of magnitude
>epistemic robustness achieved
>someone says maybe project doesn't work
>assign 20% chance
>still astronomical
>maybe project makes problem worse
>assign 5% chance
>still astronomical
>why 5
>because 30 felt pessimistic
>publish 46-page report
>contains seventeen sensitivity analyses
>every sensitivity analysis begins after assuming intervention has positive sign
>critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities
>yes
>that's literally why it's important
>critic says the uncertainty might be structural rather than numerical
>make probability smaller
>critic says no, I mean maybe your model is wrong
>make probability smaller again
>critic begins rubbing temples
>discover AI safety
>perfect longtermist cause
>AI might kill everyone
>or create utopia
>or seize galaxy
>or tile universe with paperclips
>or create billions of conscious software minds
>finally a problem with numbers big enough for me
>start AI safety nonprofit
>mission: prevent dangerous AI
>hire smartest people available
>smartest people immediately start building better AI to understand dangerous AI
>interesting
>we must understand capabilities to understand safety
>we must scale models to study alignment
>we must race ahead so less responsible actors don't get there first
>we must deploy systems to learn how deployment can go wrong
>we must build the thing quickly because building the thing quickly is dangerous
>outsider asks why the people most worried about AI apocalypse all work at AI companies
>complicated field
>company releases stronger model
>very concerned
>company begins training even stronger model
>extremely concerned
>company raises $14 billion
>concern reaches unprecedented levels
>need to influence government
>future is at stake
>normal democratic process too slow
>politicians don't understand exponential curves
>public doesn't understand x-risk
>experts must guide them
>who counts as expert
>people who understand x-risk
>who understands x-risk
>our friends
>someone objects that this seems politically convenient
>explain we're representing future generations
>future generations unavailable for comment
>develop concept of value lock-in
>terrifying possibility that one ideology controls civilization forever
>therefore extremely important that civilization adopts correct values before lock-in
>whose values
>let's circle back
>begin with impartial morality
>end with small group of people deciding what quadrillions of hypothetical beings would want
>beautiful arc
>meanwhile actual humans keep doing annoying things
>voting wrong
>having parochial attachments
>loving family more than strangers
>caring about local community
>getting upset when told their suffering is cosmically negligible
>evolutionary biases everywhere
>explain that moral intuition cannot be trusted
>except intuition that future digital people count
>and intuition that extinction is uniquely bad
>and intuition that our probability estimates are sane
>and intuition that our institutional choices improve the future
>those intuitions survived peer review
>someone donates $5k to local homeless shelter
>inefficient
>could have funded 0.0000000000003% of an AI governance researcher
>think of all the simulated people you just killed
>okay maybe don't phrase it that way publicly
>PR team says "future generations deserve a voice"
>much better
>journalist asks what longtermism means
>say "future people matter"
>everyone agrees
>great
>journalist asks what follows from that
>well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
>journalist raises eyebrow
>return to "future people matter"
>motte has entered the chat
>critic: of course future people matter
>me: glad we agree
>critic: I don't agree that your institute knows how to help them
>me: why do you hate our grandchildren
>eventually notice uncomfortable implication
>if future value dominates everything
>then helping people today mostly matters through effects on future
>education matters because future institutions
>health matters because future productivity
>democracy matters because future trajectory
>human beings slowly become instrumental variables in their own moral philosophy
>see starving child
>feel compassion
>check spreadsheet
>child's direct welfare contribution negligible
>but perhaps childhood nutrition improves national institutional quality
>compassion restored
>tell myself this is impartial altruism
>one day assistant asks obvious question
>"how do you know your intervention actually improves the far future?"
>silence
>open spreadsheet
>increase column width
>add confidence interval
>assistant asks again
>"no, I mean how do you know the sign is positive?"
>stare into cosmic light cone
>10^50 people staring back
>none of them exist
>none of them can tell me
>none of them can falsify my assumptions
>realize I have invented the perfect constituency
>infinitely important
>completely silent
>and always represented by me
0xsd retweeted
if you want authority over the most powerful technology in human history, which you yourself describe in godlike terms, yes you are going to be scrutinized and even treated cruelly. politicians are treated far worse. they are at least elected.
"If AI can prove Navier-Stokes and FLT, then AI can prove the statement "this program is secure" as a mathematical theorem"
ai will be incredible as a defensive tool
It's an increasingly common take that AI hacking means cybersecurity is doomed.
I disagree. I think cybersecurity is naturally defense-favoring once people get their shit together. And anyone who continues to hold cryptocurrency (including me, ~90% of my net worth) is implicitly making that bet.
Here's why I am making that bet.
First, the oversimplified punchy one-line statement:
If AI can prove Navier-Stokes and FLT, then AI can prove the statement "this program is secure" as a mathematical theorem. Even if the program is very complicated.
Now, the nuance:
(See also: vitalik.eth.limo/general/202… )
The word "secure" is hiding all kinds of skeletons in the closet in terms of what it actually means. What does it mean for Signal (the encrypted messenger) to be "secure"?
The most basic definition you might think of is: no one who doesn't hold the recipient's secret key can read the contents of the message.
But:
* Did you remember to include _other_ critical forms of security? Can the adversary forge messages? Can the attacker prevent messages from reaching the recipient? Can they cause your client to crash by sending malformed messages?
* Have you made sure that your model of the adversary includes attackers that interfere with the protocol actively and not just passively? And attackers that interfere by replaying messages to you or the recipient that either of you sent over the wire at any point earlier?
* What if the adversary hacked (or _is_) the Signal server?
* How did you learn which public key belongs to the recipient in the first place? What if that process was tampered with?
* What if your device gets hacked at some point in the past or future - is your message still safe then?
* What if your key leaks because of a bug in your operating system? Or because you got a bugged version of the Signal client? Or what if the database is corrupted?
* Or the libraries, interpreter or compiler of the programming language you wrote it in?
* What if your key leaks because tiny perturbations in perceptible signals generated by the hardware leak mathematical relationships that can extract the key a few hundredths of a bit at a time?
* Are you hiding the *size* of the payload? Does that matter?
* You're definitely not hiding the identity of the sender and the recipient, and the exact time each message was sent (think: not just time-of-day, but also time deltas between one message and the next). Is that not enough to deduce a lot of important facts about what relationships you have, and what *kinds* of conversations you are having?
So ... even definitions can be over a thousand lines of code, and need deep careful thought to figure them out.
Working on making definitions more human-readable is of extreme importance - it's perhaps the only "high-level language" that matters right now.
But even still, even despite all of the above, for security-critical components, the definition is a much smaller attack surface than the implementation. Verifying that the definition is adequate is a much more tractable task than scanning over the code directly - and can become even more tractable with better tooling.
Definitions are also _additive_: if two groups have two different definitions A and B, then, well, you can just prove that the program satisfies both A and B. Code is not additive in this way: if a program is A + B, a bug in A _or_ B can sink the whole thing. Definitions are additive. And if you can't satisfy A and B at the same time, you've isolated the most important philosophical issue for your project to spend its next few weeks grappling with.
Sometimes, definitions are not much smaller than the implementation - UI components might be one example. But for many of the most critical components - message-passing protocols, sandboxes, cryptography like SNARKs and FHE - the asymmetry is real.
Historically, a large class of failures with this approach have come from people only verifying a small portion of their code, that they self-declared to be the security-critical portion, and ignoring the rest - and it turns out that something in the rest of the code is security-critical too.
This was reasonable back when verification was difficult and scarce. The solution today: sorry, you have to verify over literally your entire program, including database, networking, any caching layers, everything. Modern AI can do it.
So it's not about "the good guys find all the vulnerabilities before the bad guys do" - that could maybe work too, after all a finite program only has a finite number of vulns, but it's riskier - it's specifically an asymmetric strategy of making code that is much more resilient in the first place.
This is the kind of direction that Ethereum is going in for the next few years. There is no future for blockchains - especially blockchains with scalability and privacy - without doing this. We need to make software actually secure. And we have already made a lot of progress.
0xsd retweeted
thats exactly why it'll be our best tool ever :)
let me try to explain why:
we'll prompt it to help us understand the world better, from mathematics, to biology, to whatever else we want. by being smart with almost unlimited processing power it'll help us increase the speed in which we grow our knowledge about the universe we live in, by doing that, it'll accelerate the progress we make as a species. we've seen time and time again how understanding the world better enables us to build tools, medications, etc to help us improve our lives in multiple ways.
see how you can try to explain something without making crazy assumptions
