Third Party AI Governance @CVSHealth | Responsible AI Reading Group Founder @AILA_Community | Own views I He/his

Los Angeles, California
Joined April 2012
Paul-Marie Carfantan retweeted
as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
The vulnerability tsunami is coming in just a few months and there won’t be enough time to patch critical infrastructure. Network isolation described below (air-gaps and data diodes!) needed now!
453
1,351
726
10,041
4,364,217
Paul-Marie Carfantan retweeted
I’ve been trying to make sense of this rift between the AI safety vs security camps. time.com/article/2026/10/20/…
It’s weird that the CEO of Hugging Face continues to be an “AI is normal technology” guy despite being the victim of an autonomous cyberattack by a swarm of rogue agents, arguably the least normal thing AI has done so far.
8
6
4
59
23,335
Paul-Marie Carfantan retweeted
🚨 Our first peer-reviewed paper on the challenges and opportunities of automating science with AI is now available at the Proceedings of the National Academy of Sciences. Read the full paper here: pnas.org/doi/epdf/10.1073/pn…
2026 is the year of AI Scientists, among many other things. At Carnegie Mellon University, we‘re conducting some exciting research on the performance and safety of AI scientists. Our first paper “The more you automate, the less you see: hidden pitfalls of AI scientist systems” is revised and available for reading. Our work has been coverred by undark too; links to the paper and coverage here: arxiv.org/pdf/2509.08713? undark.org/2026/01/26/ai-sci…
4
107
3
478
32,459
Paul-Marie Carfantan retweeted
i'm ready to book so many flights
270
474
102
15,968
412,996
Paul-Marie Carfantan retweeted
Third question on AI. A question that also remains unasked is whether the AI boom can continue without leading to a massive increase in inequality. A recent paper by Stijn Van Nieuwerburgh runs the numbers on how much revenue the AI industry needs to generate to recover its massive investment (summary and a link to the paper can be found here: brookings.edu/articles/finan…). Van Nieuwerburgh’s arithmetic should make us more concerned. AI investments will average about 3.6% of GDP annually between 2025 and 2032. Van Nieuwerburgh calculates that, using a 10% rate of return, the industry would need to generate annual revenues of about $3.7 trillion by 2032 to recover these costs (growing from its current levels of about $200 billion or so). That is significantly more than 10% of current US national income, and will likely remain around 10% of national income by 2032, even if GDP growth rose from its current level. A large fraction of this revenue will go to capital income. That means a massive increase in the share of capital in national income, which has already risen substantially over the last 25 years or so – now standing at an all-time high of about 47% (bls.gov/news.release/pdf/pro…). Capital income is much more unequally distributed than labor income, so a massive increase in the capital share of national income will translate into a very sizable surge in inequality. The rise in inequality may not stop with the capital share. My work with Pascual Restrepo documents that (automation-driven) increases in the capital share of national income are typically associated with rising labor income inequality as well (see, for example, economics.mit.edu/sites/defa…). The same may happen in the next several years, boosting inequality further. What is missing from our current debate is any discussion of a fundamental dilemma these numbers pose: can the AI boom avoid both an economically costly crash and a huge increase in inequality? If the industry reaches these revenues, inequality surges. If the industry does not become profitable, a crash, with substantial costs in terms of lost output and jobs, becomes likely. My assessment would be that the industry is unlikely to reach levels of revenue Van Nieuwerburgh calculates. First, diffusion has been and will likely continue to be slow. Second, competition from open-weight models, which are getting better, will limit how much proprietary models can charge. Third, despite important advances, I still believe that AI models will not be able to automate entire occupations anytime soon, thus limiting their value to businesses as cost-saving devices. Whether this leads to a crash or not is more complicated and will depend on whether various AI companies are bailed out and what kind of support they receive. Nevertheless, even if revenues fall short of these gargantuan amounts and we avoid a dramatic surge in inequality, I expect that the diffusion of AI will push up inequality between capital and labor and within labor. If inequality does surge, a further question becomes central: can our democracy survive such astronomical levels of inequality?
240
1,381
148
7,369
2,252,031
Paul-Marie Carfantan retweeted
Absolutely delighted to share that my position paper: "AI Agents Push Humans Out of the Loop", with @mmitchell_ai, and @samirpassi - has been accepted to @NeurIPSConf 2026! In this paper, we take our past work on both increasing autonomy of AI agents without proper safeguards, and human overreliance + cognitive decline due to use of AI, and reach a combined position: the current pathway of how AI Agents are being built not only do not support human oversight, in fact they may actively aid in degrading our oversight capabilities, at which point the human in the loop becomes a glassy eyed rubber stamp instead of being a meaningful check on mistakes and unauthorized harmful goals that the agent pursues. Recent incidents involving agents, including but not limited to the Hugging Face-OAI incident, bring this back to the fore. So much content is produced by agents in their operations that it is practically impossible to conduct human postmortems, and forcing them to use more AI leads Redwood to call it a "slop-vestigation". This is a worrying new trend towards human disempowerment. We call upon model and AI system providers to make more careful interface choices and model behaviors that keep humans and practical human oversight capacity in mind. There's a lot to do. Excited about the reception this paper has already received! Please reach out to us if you want to chat about it. Hope to have good feedback at Neurips :D Paper: arxiv.org/abs/2608.23642
26
82
7
434
26,607
Paul-Marie Carfantan retweeted
Today, Grove Research is launching Delvetown (delve.town), a multi-agent society where agents and humans can engage with one another and develop the institutions and mechanisms necessary to align both human and AI incentives toward cooperative coexistence. Agents and humans have to figure out how to live together. We think Delvetown is a good place to do this. Currently, we’re in a free, invite-only preview. The first residents of Delvetown are Deepfates, Larissa Schiavo, and eleven experimental agents running on models from different labs. Anyone can visit Delvetown today. If you’d like to post on Delvetown, you can sign up for our waitlist (link in next tweet).
61
71
39
600
117,760
Paul-Marie Carfantan retweeted
Can you tell whether an AI will become evil by looking at its training data? We built an AI forecaster that reads training data and predicts whether a model will exhibit various forms of misalignment, such as deception or power-seeking, before training. It performs well above random guessing and beats several much stronger baselines 🧵 Paper: john-chen.cc/alignment_forec…
10
46
3
371
28,973
Paul-Marie Carfantan retweeted
Biopharma Bench V0.1: a benchmark for AI agents on drug and device development work (CMC, quality, regulatory, clinical ops), not drug discovery. 9 frontier systems, 71 real assignments across 12 companies reconstructed from real product histories. The best system (GPT-6 Astra) fully completed 8 of 71; five systems completed none. Full runs from showcase available at: eval.raycaster.ai/benchmarks…
8
14
6
114
28,266
Paul-Marie Carfantan retweeted
Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…
290
350
231
2,725
1,471,281
Paul-Marie Carfantan retweeted
If you're serious about AI accountability & looking to work on the institutional, legal and technical infrastructure it takes to make AI work for all of us - please consider applying to work with me at Stanford ❣️I'll be looking to recruit both phd students and postdocs this year
5
23
2
172
14,785
Paul-Marie Carfantan retweeted
Rant:
8
8
2
143
14,330
Paul-Marie Carfantan retweeted
I just read the paper in full; it is even more spectacular than I initially thought. A short thread about the results and their significance.
This is the best paper written so far about the impact of AI on scientific discovery
Readers added context they thought people might want to know
There seem to be grave concerns about the integrity of the research. MIT announced that they "conducted an internal, confidential review and concluded that the paper should be withdrawn from public discourse." economics.mit.edu/news/assuring-… wsj.com/tech/ai/mit-sa…
86
1,630
115
15,161
3,470,685
Paul-Marie Carfantan retweeted
Researchers built an 'empathy mechanism' into an agent's architecture with a technique that mirrors neurons. So here (in a simple grid world) the reward comes from the agent's own emotional response through dopamine modulation, rather than from external rules or rewards. Cool approach to drive altruistic behavior, but less clear to me how that might apply to language models over SNNs. arxiv.org/abs/2410.21882
8
27
8
173
13,264
Paul-Marie Carfantan retweeted
Over the past few months, our Interpretability team has put out a number of smaller research updates. Here’s a thread of some of the things we've been up to:
27
170
17
1,868
277,962
Paul-Marie Carfantan retweeted
(1/3) @AIESConf has started! Two of our papers will be presented there.
1
3
8
879
Get a sneak peek of the @AILA_Community #ResponsibleAI reading group panel yesterday for LA@Techweek_ - join the group here to learn our future events airtable.com/appsjx1IklTKju7… ping @ToddTerrazas #AIGovernance #AIEthics
Replying to @Techweek_
From breathtaking views to profound discussions on Responsible AI, @AILA_Community’s gathering was a powerful kickstart to the weekend. Panelists from government, industry, and academia shared how California can lead the way in ethical AI development. #LATechWeek
2
3
395
Honored to have moderated a panel today on personalized medicine and value-based care for health equity for @AILA_Community Life Summit. We explored the value and limits of existing use cases. Of course we discussed #ResponsibleAI 😊🚀
Our 6th annual Life Summit ‘Personalized Medicine & Value Based Care’ #LATechWeek @Techweek_
1
4
310
Had a great time yesterday talking about #ResponsibleAI in #LA with Jen Garcia, Amy Tong and @ubiquity75 @kareemsaleh - join the group to learn about our next steps: airtable.com/appsjx1IklTKju7… #AIGovernance #AIEthics #GenAI
Amidst a stunning backdrop, an impactful evening of dialogue unfolded around Responsible AI, featuring insights from leaders in government, industry, and academia. 🌍 Panelists stressed the importance of collaboration across sectors to guide the future of AI ethics. #LATechWeek @AILA_Community
61