Beneficial AI Research, 2024 Future of Life Award For Pioneering Scholarship in Computer Ethics and AI Safety

Palo Alto, California
Joined June 2008
A group of well-known mathematicians has signed a letter decrying the impact of AI on mathematics: "A Severe Misalignment of AI in Mathematics" mathandai.org/ I'm certainly sympathetic to the changes AI is causing in an important and long-lived human discipline. But mathematics has developed what are arguably the most powerful tools for constraining powerful AI and ensuring humanity's survival. I would like to see these brilliant mathematicians applying their knowledge to that task like these two important new organizations: "Mathematical AI Safety Institute (MAISI)" maisi.org/ and the "Institute for Responsible Superintelligence (RESI)" resi.org/
1
1
15
555
My essay "Regulating AGI: From Liability to Provable Contracts" agisocialcontract.substack.c… was just posted as part of the "AGI Social Contract" project: agisocialcontract.org/
1
5
3
17
2,244
AI theorem proving advancing rapidly! Critical to AGI Safety!
Happy to introduce Kimina-Prover-72B ! Reaching 92.2% on miniF2F using Test time RL. It can solve IMO problems using more than 500 lines of Lean 4 code ! Check our blog post here: huggingface.co/blog/AI-MO/ki… And play with our demo ! demo.projectnumina.ai/
1
4
8
964
Very important work highlighting the current state of AI Cyberattacks, the likely most critical next AI Risk.
1/ 🔥 AI agents are reaching a breakthrough moment in cybersecurity. In our latest work: 🔓 CyberGym: AI agents discovered 15 zero-days in major open-source projects 💰 BountyBench: AI agents solved real-world bug bounty tasks worth tens of thousands of dollars 🤖 Autonomously. A pivotal shift is underway — AI agents can now autonomously do what only elite human hackers could before.
6
579
Extremely important work! With all the excitement around AI code generation, let's get the ball rolling on verified code generation!
1/🧵Introducing VERINA: a high-quality benchmark for verifiable code generation. As LLMs are increasingly used to generate software, we need more than just working code--We need formal guarantees of correctness. VERINA offers a rigorous and modular framework for evaluating LLMs across code, specification, and proof generation, as well as their compositions, paving the way toward trustworthy AI-generated software. 🔗 verina.io
1
4
8
928
Steve Omohundro retweeted
🧵 "What should the role of governments be after the upcoming AI economic transformation?" We're launching an new anthology on a new "AGI Social Contract" - in collab with 12+ experts to explore new strategies & policy interventions for an AI transition. In collab with: @akorinek, @degerturann, @steveom, @IasonGabriel, @JulianDJacobs, @sjmanning, @dhadfieldmenell, @sethlazar, @petersalib, @collegraphy, @synchroaphasia, @deanwball, @JustinBullock14,
11
39
10
189
43,794
Steve Omohundro retweeted
A group of leading AI scholars is backing a private governance model for AI policy solutions — and they're rallying around SB 813. 🧵
2
6
3
25
10,552
Great work! You've brought theorem proving to the thinking LLM revolution! Thank you for making the prover and the autoformalizer freely available. I believe these will be essential to effective AI Safety.
We believe formal math is the future. 🔥Introducing Kimina-Prover Preview, a Numina & @Kimi_Moonshot collaboration, the first large formal reasoning model for Lean 4, achieving 80.78% miniF2F. github.com/MoonshotAI/Kimina…
3
9
1,092
Shocking! This year's batch of Y Combinator startups is all in on "Vibe Coding": youtube.com/watch?v=IACHfKmZ… At 10:00 they say "one quarter of the founders said that more than 95% of their codebase was AI-generated!"
1
3
712
Steve Omohundro retweeted
random thoughts/predictions on where vibe coding might go: - most code will be written (generated?) by the time rich. Thus, most code will be written by kids/students rather than software engineers. This is the same trend as video, photos, and other social media - we are in the command line interface days of vibe coding. For the majority of creators, vibe coding will eventually fade, and vibe designing (with a visual paradigm) will come to dominate. People ultimately think better in a GUI-like format than a CLI-like format. Thus, in vibe designing you will show the AI the design outcomes you want, and then everything else is done for you. Yes, you may end up with tools to tweak the design details for extra controllability, and provide additional mockups that then get filled in underneath with code. But maybe folks will build software without seeing or learning a programming language. - vibe coding could reduce the need for open source libraries as more code will be generated from scratch by AI. Code will be more of a disposable commodity, with less reuse, and instead generated on the fly for personalized use. It's interesting to see right now that creating a new project is easier than editing a project, because the latter requires a lot more context/complexity. Interesting dynamics if something like this continues - "trad UX" and design standards give way to post-modern/fragmented software, as millions of new vibe coders create experiences with no prior know how and new perspectives. New patterns will emerge, as TikTok/YouTube has done to film making and trad entertainment. The world will go beyond buttons and modals and scrollbars and other things. Software may become unrecognizable before it coalesces again - if vibe coding makes software trivial to build, then the bottlenecks shift to other places: 1) consistent creativity that stays ahead of everyone else. Anyone can write a tweet, but the best creators are the ones who consistently come up with new ideas. 2) distribution and network effects, where the first vibe coded product doesn't win, but rather the first vibe coded product that hits scale that wins - imagine products that automatically adapt based on user behavior, rather than based on the actions of the vibe coder. For example, if the vibe coder has specified that the signup funnel should easy, then after seeing users struggle with it, the software can automatically vibe code itself to improve the flow by dropping steps or adding explanatory text. Right now we are in a paradigm where PMs specify behavior that software engineers specify in code. Imagine if PMs can specify outcomes, and the software is configured to automatically adapt to hit those outcomes what other wacky ideas should be on this list?
151
124
60
1,460
230,291
Amazing example of "Vibe Coding" an entire flight simulation game using thousands of prompts and no coding: xitter.cf/NicolasZu/status/18981…
Wow. I cannot believe it. Just asked Claude to make the dogfight ultra realist! ✅ hit impacts ✅ smoke when damaged ✅ explosion on death ✅ free-fall with smoke It feels so good to fly! + awesome plane and controls, 100% in Cursor with 0 code edition from me. LOOK AT THIS!
1
1
502
I just gave a short talk arguing that extending "Vibe Coding" to "Vibe Proving" and "Vibe Specification" will power formal methods for AI Safety: "It's a New Day for Formal Methods!" drive.google.com/file/d/1Jv8…
5
14
1,639
I agree that R1 shows that we are unlikely to achieve AI Safety by mandating constraints on the big lab AIs. But "Acceleration is the only way forward" is suicide. My opposite take is: "It's time to get serious about building truly safe and secure infrastructure so that humanity survives regardless of the AIs that are created."
Whether you like it or not, the future of AI will not be canned genies controlled by a "safety panel". The future of AI is democratization. Every internet rando will run not just o1, but o8, o9 on their toaster laptop. It's the tide of history that we should surf on, not swim against. Might as well start preparing now. DeepSeek just topped Chatbot Arena, my go-to vibe checker in the wild, and two other independent benchmarks that couldn't be hacked in advance (Artificial-Analysis, HLE). Last year, there were serious discussions about limiting OSS models by some compute threshold. Turns out it was nothing but our Silicon Valley hubris. It's a humbling wake-up call to us all that open science has no boundary. We need to embrace it, one way or another. Many tech folks are panicking about how much DeepSeek is able to show with so little compute budget. I see it differently - with a huge smile on my face. Why are we not happy to see *improvements* in the scaling law? DeepSeek is unequivocal proof that one can produce unit intelligence gain at 10x less cost, which means we shall get 10x more powerful AI with the compute we have today and are building tomorrow. Simple math! The AI timeline just got compressed. Here's my 2025 New Year resolution for the community: No more AGI/ASI urban myth spreading. No more fearmongering. Put our heads down and grind on code. Open source, as much as you can. Acceleration is the only way forward.
14
18
3
73
8,785
In November, I was honored to participate in the Mind First Foundation's event "AI Safety Salon with Steve Omohundro": mindfirst.foundation/present… They just posted the two videos of the event: youtube.com/watch?v=J0jxkYck… youtube.com/watch?v=ONOy5fSJ… In addition to Q&A and a panel discussion with Preston Estep and Dan Faggella, I gave a talk entitled "AI Benefits Without AGI Risks". I presented "Provably Safe AI Infrastructure" in the context of the biological evolution of agency, intelligence, and cooperation, the "Neat and Scruffy" developments in AI since 1956, and what it means for the future.
1
4
616
Steve Omohundro retweeted
AI safety people were right. Again. Instrumental convergence is proven yet again. Empirically. So far we’ve seen: ✅ Self-preservation: You can’t achieve your goals if you’re turned off (see Apollo's recent paper. Link in comments) ✅Resource acquisition: You can’t achieve your goals if you don’t have energy, money, or computing power (see Terminal of Truths making money to increase its compute) ✅Goal preservation: You can’t achieve your goals if people change your goals (See Claude below) It’s a fact. Also, it was obviously going to happen even before we got the empirical evidence, because it’s just the rational thing to do, as Claude below describes.
18
19
5
179
22,570
🏆 ⭐ We're thrilled to announce the 2024 Future of Life Award winners! This year, we honor three groundbreaking experts who laid the foundations for ethics and safety in computing and AI. Learn more about the invaluable work of Batya Friedman, James Moor, and Steve Omohundro:
5
17
3
72
24,398
PSA for Boston-area folks!! On Nov 22nd there will be a special AI safety salon at Microsoft NERD w/ Steve Omohundro (@steveom), AI safety pioneer. Organized by Alex Hoekstra & @PrestonWEstep from Mind First Foundation (@mind_first). RSVP here-> docs.google.com/forms/d/e/1F…
1
1
1
6
655