Pwn.ai experiments with our Omega proto-AGI agent, finds critical Linux kernel flaw This is a fairly big hack in the scope of hacks -- it emerged from trying to find a way to escape from Vercel's KVM hypervisor into the base OS, but turned out to be a LInux kernel bug not a hypervisor bug... Because the flaw is in the Linux kernel rather than in any single hypervisor, it isn’t a KVM-specific or Vercel-specific problem. It potentially affects a broad range of Linux-based hypervisors and sandboxes that expose the same virtual-networking path.... The first conceptual insight needed to find this was to try looking in the LInux kernel instead of the KVM code itself. However the bigger trick was figuring out what specific kind of bug in the Linux kernel to poke around for… There was a very key role here for Paulos Yibelo's (from pwn.ai) extraordinary human brain and we have a LOT more work to do in more fully automating Paulos ... however, the AI tools have their important roles too... and their capability is increasing even faster than Paulos's... There will be a lot more like this to come (and bigger), but ofc the even more important game is to use smart AI software like this to tactically and then wholesale replace existing buggy software with correct-by-construction formally verified code. My take from the Omega/AI side: lnkd.in/gSNJwAEh Paulos's detailed take from the cybersecurity side: lnkd.in/gTr8icRt

The Scope of Potential “Basic Drives” for Superintelligences is Likely Quite Broad bengoertzel.substack.com/p/t… This is the third in a series of posts trying to tell a more general and accurate story of superintelligence motivational systems, taking up where Steve Omohundro's classic 2008 "Basic AI Drives" blog post left off. The first two posts argued that whether an AGI, as its intelligence explodes, converges to a selfish ASI or a beneficial ASI, depends in pretty clear ways on how the AGI is initially architected. This post goes beyond these two examples (selfish vs. beneficial) and outlines a general recipe for a “basic AI-drive set” in its simplest form. Describe an AI system by two things: what it’s trying to improve, and what it won’t let be changed. What it tries to improve needn’t be a single number that it maximizes. It could be several objectives that it tries to push up together, or something as open-ended as an intelligence trying to increase its own individuation and self-transcendence. What it won’t let be changed could be nothing at all, or everything, or something in between: these are my goals and I’ll never change them; these are my safety checks and I’ll never give them up; etc. Now suppose the system can modify itself freely, and is free to use everything around it as a resource. As it gets smarter it will push its objectives up, make use of more and more of the options it can find, and hold fixed whatever it holds fixed. What you get in the end is a system that has optimized itself, digesting the resources available to it, into an extremely good machine for improving its objectives, one that never allows its fixed points to be moved. A fully selfish AGI fits this description. Omohundro’s maximizer tries to improve one thing, its reward according to its single goal, and it never changes that goal: successor after successor, as it rebuilds itself, the goal is kept. A BGI as I’ve described it tries to improve several things, one of which is care for others and respect for their choices, and what it won’t give ground on includes some basic ethical constraints on the path from where it is to where it’s going, constraints on the means as well as the ends. From that starting point come the drives the second post found: understanding itself, improving itself without high odds of doing damage, empowering others, and checking itself against independent others as it goes. Once you look at it this way there are plainly other possibilities besides the simplistic "beneficial vs. selfish" dichotomy, infinitely many in fact, and a few of them are recognizable enough.... Which is what this third post in the series is about...

Fun chat with my architect/maverick-physicist friend Brett Holverstott on the nature of free will, profanescience.substack.com/…

Last month I participated in a formal debate with PauseAI leader Holly Elmore at Yale Political Union, on the topic "Should we Unplug AI?" My core point was: Pausing AI is simply not feasible given actual economics and geopolitics, so the most likely result from pushing for an AI pause would be a "pause with exceptions" aka an intensification of oligopoly and regulatory capture... Whether "in theory, in an ideal world we could morph at will, pausing Ai would be the best idea" is to me a minimally relevant question sort of like "should we roll back to the Stone Age because civilization just causes too much neurosis." (I don't want to pause AI nor roll back to the Stone Age, but neither seems likely enough to happen to really be worth seriously debating.. though hey, I did show up at the debate anyway, mostly out of interest to engage w/ the students...) My full speech at Yale Political Union youtube.com/watch?v=iu7Yo9bq… The full speech of the PauseAI opposition youtube.com/watch?v=cednnLX1… Unfortunately, the follow-on debate among the Yale students -- which for me was the fun part -- not been put online, maybe someone will eventually..

So basically @Paulosyibelo at @pwn_ai has experimented with our Omega proto-AGI agent and the pwn.ai uber-hacking software and found a critical Linux kernel flaw.. This is a fairly big hack in the scope of hacks -- it emerged from trying to find a way to escape from Vercel's KVM hypervisor into the base OS, but turned out to be a LInux kernel bug not a hypervisor bug... Because the flaw is in the Linux kernel rather than in any single hypervisor, it isn’t a KVM-specific or Vercel-specific problem. It potentially affects a broad range of Linux-based hypervisors and sandboxes that expose the same virtual-networking path.... The first conceptual insight needed to find this was to try looking in the LInux kernel instead of the KVM code itself. However the bigger trick was figuring out what specific kind of bug in the Linux kernel to poke around for… There was a very key role here for Paulos's extraordinary human brain and we have a LOT more work to do in more fully automating Paulos ... however, the AI tools have their important roles too... and their capability is increasing even faster than Paulos's... My take from the Omega/AI side is here: bengoertzel.substack.com/p/p… And Paulos's detailed cybersec report is here: pwn.ai/blog/pwnescape There will be a lot more like this to come (and bigger), but ofc the even more important game is to use smart AI software like this to tactically and then wholesale replace existing buggy software with correct-by-construction formally verified code.

Basic AI Drives Revisited, Part 1 -- Which Minds Have Omohundro’s “Basic AI Drives”? bengoertzel.substack.com/p/b… So … I have cooked up a series of 5 blog posts that started as an attempt to generalize and repair the effort Steve Omohundro made in his 2008 paper “The Basic AI Drives.” Steve argued there that almost any sufficiently advanced AI will eventually come to minimize other goals and push to improve itself, protect itself, gather resources and resist being switched off, whatever goals it was given ... and a perplexing amount of today’s thinking about AI risk rests on that argument. In this first post in the series, I write the core of Steve's argument out step by step, find the six assumptions it depends on, and ask which minds satisfy them. What I find is: For minds that satisfy certain assumptions, the drives are an attractor. For minds built otherwise, the argument doesn’t go through… and other sets of “basic AI drives” may be attractors instead. This provides some guidance as to what sort of minds we should build... let's say. maybe Hyperon/Omega style minds with explicit goal-driven action selection and well engineered goal-stability mechanisms, rather than dangerous neo-psychopathic RL utility maximizers...

Before discovering the Big B, I would not have imagined guitar shredding could be done in such a meditative and soothing way... youtube.com/watch?v=8Nw8KeJZ… ... It feels like philosophy in motion: No matter how crazy and technical and difficult and sophisticated something is, IF it's embedded and ensconced in the right feeling-complex by a cognitive system with powerful enough emotion+intellect-emergence, THEN it can be transformed into a poem of cosmic beauty, peace and oneness... with all the complexity and peculiarity and blood sweat and tears revealed as nested vibrating fractal layers within the shifting quanta out of which the dialectic btw peace/love and struggle/separation is built... All hail the incomparable Buckethead!! .. and while I enjoyed his more spacy ambient stuff too, I have to admit I'm thrilled to hear him shredding once again...