Red-teaming innovation, fine-tuning regulation ——— AI policy @OpenAI
Paris
Joined June 2016
- Tweets2.6K
- Following1.5K
- Followers2.6K
- Likes13.2K
Martin Signoux retweeted
Huge: The Chinese government moves to place registration and licensing requirements on frontier AI model weights, chip designs, and other key AI technologies.
A registration requirement is effective immediately, with a public comment period over the next month before the more expansive regulations are finalized.
The draft includes a range of prohibitions and licensing requirements that could shape the evolution of the Chinese AI ecosystem. It also provides simplified pathways for certain open-source releases and a range of excluded areas.
This is a major step and could give Beijing substantial new leverage over the global spread of Chinese AI. But there are lots of questions about the practical consequences:
—How much will model companies reshape the definitions, thresholds, and processes during the "consultation" period?
—What will firms actually start doing to comply in the coming days?
—Will implementation materially restrict the global spread of Chinese models, or chiefly give Beijing greater visibility into—and discretion over—how they spread?
MofCom & MIIT just confirmed our earlier scoop with a full update of export controls on AI, semiconductor, healthcare etc : google.com/url?q=https://www…
Martin Signoux retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
Martin Signoux retweeted
Announcing our work textGrain, @OpenAI's text watermark in response to the EU AI Act. OpenAI is rolling out textGrain in the EU and opening detector access to researchers.
We'll open-source it so the community can build on top.
We're expanding our approach to content provenance to include text in response to EU regulatory requirements, while recognizing the significant limitations of current text watermarking technology.
Our tools already help verify whether an image or audio file was created with our models. This work builds on those efforts to help people better understand when content may have been generated or edited with an OpenAI model.
In the EU, we’ll start watermarking eligible text from ChatGPT and Codex over the coming weeks to comply with the EU AI Act.
Customers using our API can turn on text watermarking for select models worldwide today.
Martin Signoux retweeted
I agree with this. These web data access incidents tell us something about the scale and persistence of agents, but are not sophisticated from a cyber perspective, so using the word "hack" is not helpful.
To be quite frank, a lot (not all!) of these “rogue agent hacks” on government statistical websites are things that think-tank interns and research assistants have done for many years. I have known many a think tank paper that was enriched by an enterprising RA using eg urlquery to find CSVs and PDFs on government agency websites that were publicly accessible and non-sensitive but nonetheless not things the agency *intended* to be on their website. Is it “hacking” to find those things? Is it hacking in the even more innocuous examples where agents are literally just accessing tabular data available on a government statistical website, just a version of that data that is easier to access and analyze programmatically?
This constant trickle of examples (which tbc I have personally seen many models do, before I joined OpenAI; it is even something I have discussed with models before!) will probably have the effect of diminishing the importance of “an AI hack” on the eyes of the observing public by creating an oversupply of “AI hack” examples. OAI-HF was an example of AI hacking. Many of these more recent examples, very candidly, are straining the definition of the word “hack” and, I worry, cheapen the non-technical public’s understanding of that concept.
Martin Signoux retweeted
I'm excited for safety cases as a north star. Formal safety arguments are a great tool for highlighting residual risks and enabling risk-informed model development decisions
openai.com/index/towards-saf…
Martin Signoux retweeted
We have just finished the inaugural meeting of the Rhine Group. We discussed the central elements of Europe's competitiveness agenda, including basic science, AI/compute and cybersecurity, defense, energy, entrepreneurship/start-ups.
I will be updating you on any progress we may have. Some of the projects we will pursue include:
- We'll help Europe catch up on compute (Mario Draghi diagnosed the problem and some solutions in a recent FT piece)
- We’ll help European universities and scientific institutions and explore a Rhine Chair program with European universities.
- We’ll work on some key EU policy challenges, including facilitating the flow of capital to entrepreneurs.
- We will set up a grants program that I will explain and announce here.
Martin Signoux retweeted
AI is getting cheaper more quickly than any other transformative tech in history. At a given level of performance, cost has fallen ~47%/quarter since 2023.
That’s 4× faster than DNA sequencing, 6× faster than compute, 18× faster than lithium batteries, and (up to 1973) 54× faster than electricity.
Martin Signoux retweeted
Today’s blog post outlines our priorities for this work: safety cases, capability and alignment evaluations, safeguard testing, and independent investigations of critical misalignment incidents.
openai.com/index/priorities-…
Martin Signoux retweeted
Great thread on France's productivity slowdown!
Martin Signoux retweeted
I was encouraged this week to see the leaders of the frontier labs agree on the need for them to slow down the pace of AI development. Given the stakes, it’s a good and necessary first step.
But I’m even more encouraged by the growing recognition that how this powerful new technology develops should be at the center of our public debate.
I’ve been watching the progress on AI for over a decade now, and one thing that’s clear to me is that the potential impact of this technology is not overhyped. It’s also moving at lightning speed – and even faster than those who are engineering it can keep up with.
I’m not an AI accelerationist who believes it will lead to some techno-utopia, and I’m not a doomer who thinks it will inevitably lead to humanity’s destruction.
But whether this technology results in amazing breakthroughs in medicine, energy and education or unleashes huge economic disruptions, greater inequality, and potential catastrophe will depend on the choices that we make right now – choices that should be made not just by the companies involved, but by all of us.
Martin Signoux retweeted
About a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the foundation for ARC 4 and ARC 5.
We're incredibly excited to share what we've been building. We're still on track to release ARC 4 in Q1 next year, as promised.
ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity.
Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology.
Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it.
Open source is the foundation for that progress.
The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.
Martin Signoux retweeted
In 3.5 years, we went from GPT-4 which couldn't reliably add two numbers together to a 10,000 agent system that can autonomously solve a Millennium Prize problem. This is what exponential progress looks like. Accelerating, awe inspiring and terrifying.
Martin Signoux retweeted
An effect of the rapid acceleration of AI is we are losing an empirical handle on what is happening in the actual micro-processes of work in the post-2026 long-running agentic era.
We have some academic knowledge of the impact of chatbots on work, much less on impacts of agents.
Martin Signoux retweeted
Seems like test time scaling has gained a 3rd axis: latent space reasoning iterations in looped transformers.
Martin Signoux retweeted
Alongside the launch of GPT-6 Astra, we're committing $1 billion to subsidize Daybreak access and frontier capabilities to help frontline defenders protect essential services and critical infrastructure.
Welcome to the AGI era for cybersecurity.
openai.com/index/daybreak-fo…
Martin Signoux retweeted
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
Martin Signoux retweeted
An announcement: Under the leadership of Mario Draghi and @patrickc we have set up the Rhine Group: policymakers, economists, entrepreneurs and business people pushing European reforms and the Draghi agenda.
rhinegroup.eu/
Martin Signoux retweeted
1/4 Everyone is talking about the #futureofwork and AI. This is why we recently launched The Economics of AI, a new Substack looking at how AI is reshaping work, productivity, and economic opportunity.
In our latest piece, we look at an idea that gets little attention in conversations about the “future of work”: the care economy.
A few takeaways:
Martin Signoux retweeted
Pleased to share the launch post of the AI Futures blog at OpenAI! I won’t be the sole contributor to this blog, but I’ve written the first post, and it’s all about the most challenging risk area in all AI policy: concentration of power.
Martin Signoux retweeted
As models become more capable, the risks associated with developing and testing them internally also grow.
We temporarily paused reinforcement learning (RL) training on our latest models intended for deployment for two weeks while we hardened and red-teamed our research environments and expanded monitoring coverage.
Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations validate these safeguards and establish more evidence of alignment.
openai.com/index/pacing-mode…
