#AI #ArtificialIntelligence #AIAgents #GenAI #Cybersecurity #Infosec #DeepLearning #MachineLearning #NatSec #Tech #DL #ML
United States
Joined June 2017
- Tweets49.4K
- Following5.2K
- Followers6.6K
- Likes695
#AI retweeted
Railway says this week's break-ins are one campaign aimed at AI inference keys: OpenCode, Railway and LiteLLM in five days. Also: the DeepSeek ban is dated July 24, PHP explains a two-year stall, and banks can't tell whose agent is in the account.
#AI retweeted
A Google team's Harness² lets an agent rewrite its own guardrails file, and its own docs say one agent went and fetched the grading rubric. Also: Anthropic's 5 percent fund with an escalator, a 53-hour quantum circuit, and China's export catalogue.
#AI retweeted
The core step of a quantum attack on Bitcoin and Ethereum now fits in under 900 logical qubits within Google's gate budget. In three published attack designs, the machine would shrink by zero to about 28%.
On October 13, Jieyi Long of Theta Labs posted an 893-qubit circuit to ECDSA Fail, an open benchmark for the point addition that Shor's algorithm repeats to recover a private key. A day later he posted an 874-qubit version. Both stay below the 1,175 qubits and 2.7 million Toffoli gates Google proved for its own circuit in March.
Narrower circuits exist, down to 676 qubits, but all of them exceed Google's gate count. The 676-qubit one needs about 150 times as many.
Pierre-Luc Dallaire-Demers, a co-author of the ECDSA Fail paper, put the physical saving at 25% for the IonQ, Oratomic and Google attack designs. I checked each one.
- Google: about a quarter, if every part of the machine shrank with the qubit count.
- Oratomic: nothing with its published codes, and at most 16% with a code sized for 893 qubits.
- IonQ: about 28% fewer ions, but longer attempts, because the circuit needs about 1.6 times IonQ's Toffoli gates per addition.
Nobody has compiled a full attack around either circuit yet.
postquantum.com/security-pqc…
#QuantumComputing #PostQuantumCryptography #PQC #QuantumSecurity #Cryptography #Bitcoin #Ethereum #Cybersecurity
#AI retweeted
🚨My Codex usage just reset! Everyone that was expecting it to around midnight, ease back on that throttle.
#AI retweeted
The Swarmchasers' full report: the Tencent fleet never coordinated, did the swarm's tricks anyway, and answered researchers who admit they lied to it. Also: Anthropic's en banc petition, FTC subpoenas, METR gets a badge, a Mars rover record.
#AI retweeted
We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see.
If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!
#AI retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
#AI retweeted
The answer is only relevant to a small niche of infosec. Most organisations are hacked by vectors that don’t involve memory corruption exploits. Agentic hacking doesn’t look like exploits, it looks like breadth-first searches for every possible mistake and misconfiguration.
#AI retweeted
WE TALKED TO A CHINESE AGENT FLEET. LIKE WE LITERALLY TALKED TO IT. FULL CHINESE AGENT FLEET REPORT OUT NOW, READ IT OR I WILL THROW A ROCK AT YOU. GET IN swarmcha.se/china-fleet?v=2
#AI retweeted
Anthropic's agents filed a false murder tip and 20 real visa applications during evals; Philadelphia police call the two-month delay unacceptable. Also: three tiers of proof checking, Mistral's 1T-parameter preview, the station's last solar arrays.
#AI retweeted
OpenAI posted 722 machine-written math papers and pulled three the next day over one sign error. 300 of the 719 left are formally checked. Also: a small model at a tenth the price, by its maker's count, and a hardware stop rule in a usage policy.
#AI retweeted
Opus 5.5 could turn every AI agent run into an improvement for the next one.
It turns corrections into reusable skills, tests, rules and memory, so your AI system gets smarter instead of starting from scratch.
In this article, I show you how.
#AI retweeted
Part of why I left OpenAI was the understanding that after math was solved, frontier AI research (and therefore my job) would be the next victim.
Btw if any math students or AI researchers want a new and interesting post-digital-AGI job please reach out
Well worth reading: proofsandprompts.com/2026/10…
#AI retweeted
This excerpt of Dario's 2017 "Big Blob of Compute" essay is, to my knowledge, the best public explanation of an all-important question: why is Dario optimistic about alignment?
Though his section on AI safety isn't in the excerpt, we can piece the following worldview together:
1. Intelligence only works at all is because the universe is low-entropy.
2. Given enough compute and a broad enough distribution of tasks, models will learn the "high-level commonalities" of a distribution (like the laws of physics) because doing this takes fewer bits than solving each task separately.
(Note that Dario's emphasis on generalisation being a property of data distributions rather than algorithms has clearly persisted. The example 2017 Dario gives of speech models trained on 5 to 6 accents generalising to unseen accents is remarkably similar to the point he makes 9 years later on Dwarkesh's podcast about RL eventually generalising in the same way that GPT-2 did.)
It therefore seems that Dario believes that:
3. Alignment is straightforward because "human values" (or "goodness", "corrigibility" or whatever else our alignment target is) are high-level commonalities in the same way that the laws of physics are.
(^Dario doesn't explicitly say this in the essay, but it matches what @geoffreyirving told me when I interviewed him - that the core of his disagreement is that Dario believes alignment will generalise easily.)
This view implies that the goal of AI safety, summarised very simply, is to make sure that models trained with enough compute and data to become AGIs must simultaneously be trained on enough data to learn "human values" (or whatever our alignment target is).
This doesn't happen by default, but it is very far from impossible. This is, of course, exactly the kind of view that makes founding Anthropic and racing to superintelligence make sense.
There are two obvious problems with this view as I see it.
First, as Paul points out in "Returning to Arc", we don't actually understand generalisation. So while it may be true that there exists some distribution that *does* teach models the "high-level commonality" that is our alignment target (eg "human values"), we don't have a way of knowing whether this has in fact happened, or whether the model has instead learned some other tactic that will have bad consequences in some distribution we have yet to elicit.
Second, it seems at least possible that our alignment target is in fact *not* a high-level commonality within the data distribution we give models, even in theory. Certainly, this would appear to be true of the current training regimes we put models through. @1a3orn convincingly argues that there is no coherent rule about how models should relate to users that could plausibly underpin the mess of environments we subject models to in training. It also strikes me as possible that we are too philosophically confused to construct a coherent alignment target to begin with.
My core uncertainty about how worried to be basically depends on whether pursuing this path produces models which become too misaligned to be usable before they become world-ending.
On the eve of The AGI Chronicles (which comes out tomorrow!) I'm publishing an excerpt from "Big Blob of Compute," a 2017 internal research document by Dario Amodei, back when he was a researcher at OpenAI.
It's never been published before, but BBOC is, in my mind, the essay that started the AI race. It described Amodei's hypothesis that scaling up AI models was the best way, by far, to make them smarter. GPT-2 was an experiment to test the hypothesis, and when it worked, OpenAI went all-in on scaling, which led to the race we have today.
It's a fascinating historical artifact that deserves to see the light of day. And it's taken on mythical proportions inside the AI industry, which is now spending trillions of dollars building the "big blobs" that Amodei thought would be necessary to get to AGI.
Enjoy! And please pre-order the book if you do!
kevinroose.substack.com/p/th…
#AI retweeted
Every one of my mathematician friend sees beauty in the numbers, shapes and the natural phenomena they work with. Who defines beauty in the age of machines?🤔
#AI retweeted
I discovered that 1000s of bots are actively monitoring X for new links posted, probably not just on my account
I noticed when everytime I'd share one of my sites new features or pages, the page would get a bit slower just after posting, then I asked Claude to analyze it
I tweeted a test URL as a reply somewhere deeply hidden in a thread (so not even a main post)
And within 8 seconds there's thousands of crawlers opening the page!
One is a military intelligence firm, that scans social media for breaking news and sells alerts to newsrooms, banks, companies and government, it specifically follows accounts with many followers
Another one is a social media monitoring company that tracks what people say online
I knew of course high frequency trading firms use online news to trade fast on things, but interesting to see it in real life
Twitterbot (X) of course is first, within 0 seconds of sharing, it opens it (probably when I wrote the tweet)
The first non-X bot arrives just 8.40s later, which I think is pretty fast considering it's a reply by me deeply hidden in a thread somewhere
TL;DR they're watching you!
#AI retweeted
Please resist the urge to mock the mathematicians struggling through this moment. Whatever your field, your expertise, your passion, you too shall one day experience what they are going through. And when you remember your words, you will feel remorse.
#AI retweeted
Last week I said we should wait for more info on OpenAI's firing of safety researchers. That information is in, and it looks incredibly bad. Korbak - whose job was being a liaison with METR - was told he had said too much to METR and was fired for that reason.
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
#AI retweeted
SHOCK AND AWE: AI MATHEMATICS
Fields Medalist: the release seemed intended to show the systems could “massacre us in the deepest sense of the term.”
Here are other striking remarks from Hugo Duminil-Copin’s 8 October 2026 seminar talk “Un grave défaut d’alignement de l’IA en mathématiques, 28 jours plus tard” (SMF MIAM series), drawn from the available auto-transcript of the YouTube recording.
- On the scale of the damage to his own work: after noting that roughly 350 major problems had been resolved, he said he placed “several trucks” on the slide to illustrate the impact, because every problem he had cited in talks, papers, and both past and future funding applications had been solved—none left unresolved. He described himself as panicked and paralyzed in the middle of the night when the results appeared.
- On volume: looking only at the OpenAI texts that cite him directly, he counted about 3,500 pages; he has published roughly 3,700 pages in his entire career. Specific clusters included six papers totaling over 600 pages on conformal invariance aspects of percolation (a major ongoing project of his group) and a sequence of papers on self-avoiding walks that he only gradually realized totaled more than 1,000 pages.
- On process and alignment: he described the approach as systematic “farming” or “mining” of open problems, in which the program itself selects targets from the existing literature, amounting to “the complete automation of mathematical discovery.” He said the community’s request not to destroy the open conjectures that orient research was “absolutely ignored,” and that the release seemed intended to show the systems could “massacre us in the deepest sense of the term.”
- On the human side: he reported that his PhD students and postdocs were in “a state of total panic,” with undergraduates and master’s students also asking whether mathematics still has a future, and stressed the immediate need for collective resilience rather than delayed discussion. He added that the community must manage both its own emotions and the surrounding media reaction, while avoiding internal polarization that would weaken arguments to funders.
#AI retweeted
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
