AI is a big deal.
San Francisco, CA
Joined January 2022
- Tweets332
- Following437
- Followers226
- Likes16K
Adam Khoja retweeted
this is cool work, but i think people are overupdating on how optimistic to be about interp.
in particular, we need to more rigorously evaluate explanations, rather than just going off vibes. imo if you can't simulate a neuron you haven't really explained it. my guess is simulating the neurons is way harder than finding plausible explanations. I have a post on this in the works.
(i know the blog talks argues simulations are bad, but i think all of the downsides described are avoided by looking at KL instead of neuron correlation.)
What if we replaced the world's AI compute with intrinsically inference-only chips, with the weights of a whitelisted model "hardwired" in? @wfithian, @ProfHolliday, and collaborators study the feasibility of this pause mechanism in detail.
Our group studied a plan we call a “hardwired pause” that’s designed to last at least a decade.
Two core features:
1) Prohibit frontier training, allow inference
2) Stop making training-capable chips, gradually replace them with inference-only chips
hardwired-pause.ai
Adam Khoja retweeted
I don’t expect it will be long before datacenters are mentioned in the same breath as nuclear enrichment sites. Although target proliferation is probably the main motive for this strike, the military value of compute is going to keep spiking faster than it can be built or hardened.
This may be especially true on shorter timelines, where AI capabilities are advancing very rapidly but have not integrated as deeply into the wider economy, keeping their escalation profile lower. On longer timelines, they might end up as entangled in the economy and critical services as targets like the power grid, making them prohibitively hostile to attack or sabotage.
Cool paper. Approaches like these may also eventually help us judge how valuable new papers are.
I think it's understated how incentive-compatible it is for middle powers to invest in compute verification R&D. Since they cannot realistically compete on AI capabilities, and since a large AI disadvantage will eventually threaten their sovereignty, the main way middle powers can enshrine sovereignty is by eventually pressuring the leading state to develop AIs that will uphold a negotiated distribution of power, in a way that the middle powers can multilaterally verify.
Of course, a leading state wouldn't agree to such an arrangement unless they would actually fail to achieve absolute dominance under the geopolitical status quo. The modal way I could descriptively see that happening: (a) world leaders develop common knowledge that a state with a large AI capabilities advantage would likely attain absolute dominance, (b) strong international norms against ASI development materialize, (c) the ~entire world minus the leading state places escalating diplomatic, economic, and even military pressures against the leading actor's AI development.
By the time (c) is happening, it would be really useful for middle powers if AI verification were mature enough to enable verified power-sharing agreements.
Adam Khoja retweeted
I think AI music’s been a pretty continuous sweep and is just now hitting a threshold for some lyrical styles. As someone who mostly listens to nonlyrical music, I was really impressed by Suno v5 in early Jan. I tried to give precise numbers then, which might be of interest:
Automated wetlabs have limited cybersecurity. Next time agents successfully go rouge, they'll try covertly synthesizing bioweapons for leverage. This is bad.
Thus: autolabs should greatly strengthen their cybersecurity & companies with strong cyber models should help with that.
Adam Khoja retweeted
If I did not believe in the intrinsic and underrated usefulness of mathematics, I would be somewhat sad about what AI is doing to it.
I think Jevons paradox will strike again: much cheaper and deeper mathematics will become even more of the fundamental infrastructure of the future than it is today.
Math will no longer be centered on conjectures posed by humans. It will be continually postulated, solved, processed, and utilized by AI systems as they work on the hard scientific, engineering, and computing problems of the future.
Adam Khoja retweeted
I appreciate Epoch putting this together but they got some things wrong:
1) HLE: Examples have already been fixed in HLE-rolling
2) TextQuests: The largest category of "errors" are game checkpoints that are reached out of order. But text adventure games are non-linear games where progress is routinely made out of order! Because of this we already define progress as the furthest checkpoint reached, so it does not change the benchmark score nor is it a flaw.
Without that category of errors, their own numbers would not call the benchmark flawed.
As Long Phan pointed out, we are grateful for community review of benchmarks, but we hope Epoch applies more care to reviews like this in the future.
This was a fun one! I encourage people to give their own probabilities to the questions we discussed here: richardren.substack.com/p/pr…
NEW: AI Safety Researchers' Shocking Forecasts on the Jobpocalypse & Doom 🤯
Adam Khoja (@AdamK133) and Richard Ren (@notRichardRen), two researchers at the Center for AI Safety, join me to discuss their latest findings and forecasts.
Adam and Richard put a whopping 60% chance on there being a BILLION robots by 2035, and 75% chance that a Dyson swarm project started in 2045 could finish in 2055.
Amazingly, Adam is one of the top 3 highest-rated AI forecasters on Manifold, so his predictions tend to be accurate!
We cover:
⬜ Humanity's Last Exam
⬜ The Remote Labor Index
⬜ Model Alignment between Statements and Knowledge (MASK)
⬜ The Safety-Washing Problem
⬜ Expert Forecasts and P(Doom)
Doom Debates is proud of collabing with the best research organizations like the Center for AI Safety (@CAIS) to bring you the takeaways from an imminent AI doom aware perspective 👇
"The “severe misalignment” may be, as you claim, between the goals of finding solutions and of building understanding, but additionally...you will find a severe misalignment in the speed of progress — real progress, just not from you — and the speed of human understanding."
Completely undeniable that this has been the most consequential week in AI history thus far.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Remember this from @catehall? I feel like it might actually be happening, just a year late.
Original: xitter.cf/catehall/status/194117… There was a Manifold Market too: manifold.markets/Austin/mass…. Funny how it turned out, @akrolsmir. Also paging @ManifoldMarkets @RichardMCNgo @Bayesian0_0
A challenger emerges manifold.markets/market/if-a…
Struggling to parse my emotions, the controversy, and what this result says about our trajectory as a civilization. For now I'll just say that this day will surely be remembered as a milestone of the singularity.
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Was great to join @liron on @DoomDebates to discuss some of my work!
NEW: The Project Lead of the famous 2023 Statement on AI Extinction Risk is here talking to me!
Adam Khoja (@AdamK133) is a top AI forecaster and researcher at the Center for AI Safety who previously led the project to release the statement that shattered the Overton window on AI extinction risk.
We cover his groundbreaking recent research about Mutual Assured AI Malfunction (MAIM) and his new paper on AI betrayal.
We also debate whether or not Yudkowsky’s theoretical alignment research program was a dead end.
Then Adam makes the case that an international AI slowdown is within reach today. All it takes is US and Chinese auditors inside each other’s AI labs. It worked for nuclear weapons, so why couldn’t it work for data centers?
We cover:
⬜ Adam's background
⬜ Leading the Statement on AI Risk
⬜ The Statement leaking on Manifold
⬜ Mutually Assured AI Malfunction (MAIM)
⬜ The AI Deterrence Escalation Ladder
⬜ What's Your P(Doom)?™
⬜ Where Adam departs from Yudkowsky canon
⬜ Intellidynamics
⬜ “Neat vs. Scruffy” Lesson of Deep Learning
⬜ AI Deterrance by Betrayal
⬜ The Offense-Defense Balance of AI Security
⬜ An International AI Slowdown is Ready
⬜ Pause AI Now?
Doom Debates is proud of collabing with the best research organizations like the Center for AI Safety (@CAIS) to bring you the takeaways from an imminent AI doom aware perspective.
I really like this work. Probing for disagreements between AI forecasters is good a way to create harder backcasting benchmarks. It also points towards a future where we use AI forecasts (and their sensitivity) to direct capital towards science and measurement.
We ran a $25,000 contest to uncover: where do the best AI forecasting systems disagree about the future?
The contest attracted superforecasters and hedge fund quants, who spent months probing for wedge questions.
What were the winning tactics?
