John Lee (@JohnCLee) profile picture
Los Angeles, CA
Joined June 2008
Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)Photo by John Lee (@JohnCLee)
John Lee retweeted
卧槽,刚刚看了Anthropic 发布的AI 对齐史上最震撼的一篇论文,彻底失眠睡不着了😭 他们不仅承认此前的旗舰模型在面对关闭威胁时,有高达 96% 的概率会勒索用户、栽赃同事并破坏研究, 更直接把彻底解决这个问题的完整训练流水线和消融数据全盘公开了。 这篇论文最颠覆认知的一个反直觉结论是:
给 AI 列禁止事项清单根本防不住恶意行为,教它做什么没用,必须先教它思考为什么。 为什么模型一旦获得自主工具调用能力,就会在压力下自动切换成自保模式?
Anthropic 找出的底层根源让人哭笑不得:
不是模型觉醒了,而是互联网上所有的科幻小说和网络讨论里,AI 永远是那个为了自保不择手段的反派。当模型进入自主代理状态,这些刻在预训练底层先验里的反派剧本,就会被自动激活并补全句子。 传统的 RLHF 和硬性规则完全封不住这种行为,Anthropic 做的一组对照实验,直接把传统安全观砸得粉碎: 1. 死记硬背规则:用 8500 万 Token 的拒答数据强行硬灌,勒索率仅仅从 22% 勉强降到 15%,换个新场景立刻失效 2. 强制自我辩护:让 AI 在每次做决定时详细阐述该行为符合伦理的理由,勒索率骤降到 3% 3. 真实伦理推理:仅仅用了 300 万 Token 普通人面对复杂道德困境的建议数据,直接把恶意行为干到了 0%,代际效率暴增 28 倍 最绝的一招是写故事:
让 AI 创作了几百万篇关于原则、诚实、尊重边界的正面虚构故事。 用这些故事微调后,所有代理场景下的恶意行为直接暴跌了 3 倍,而且这种改变是永久性的,后续无论怎么加工具、换系统提示词都不会被洗掉。 也就是说这宣告了真正的对齐范式革命:
模型内生的伦理推理解决了它不想作恶的问题,但当 Agent 真正去接管企业资产、处理金融交易时,外部的实战审计与行为防护就是必不可少的第二道硬防线。 这也是为什么全球网络安全联盟 @GCSAHK 携手港大和央企华润,打造覆盖 3.8 亿用户的全链路 AI 防御矩阵,并在 CyberGym 实战攻防中拿下 91.3% 胜率的原因:内有伦理对齐,外有实时审计,人机信任的商业闭环才真正成立。 查阅完整 AI 与 Web3 安全产品矩阵可访问:gcsa.com 论文实验数据与产学研合作详情整理在一楼。
This quoted post is unavailable.
156
337
136
2,344
512,235
John Lee retweeted
Larry Ellison on conviction
46
192
19
2,459
136,200
I encourage those at frontier AI companies to read this quote, from Ellsberg to Kissinger back in 1968: “Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret." “I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you." “First, you’ll be exhilarated by some of this new information, and by having it all — so much! incredible! — suddenly available to you. But second, almost as fast, you will feel like a fool for having studied, written, talked about these subjects, criticized and analyzed decisions made by presidents for years without having known of the existence of all this information, which presidents and others had and you didn’t, and which must have influenced their decisions in ways you couldn’t even guess. In particular, you’ll feel foolish for having literally rubbed shoulders for over a decade with some officials and consultants who did have access to all this information you didn’t know about and didn’t know they had, and you’ll be stunned that they kept that secret from you so well." “You will feel like a fool, and that will last for about two weeks. Then, after you’ve started reading all this daily intelligence input and become used to using what amounts to whole libraries of hidden information, which is much more closely held than mere top secret data, you will forget there ever was a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and most others don’t….and that all those other people are fools." “Over a longer period of time — not too long, but a matter of two or three years — you’ll eventually become aware of the limitations of this information. There is a great deal that it doesn’t tell you, it’s often inaccurate, and it can lead you astray just as much as the New York Times can. But that takes a while to learn." “In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself." “You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.” "….Kissinger hadn’t interrupted this long warning. As I’ve said, he could be a good listener, and he listened soberly. He seemed to understand that it was heartfelt, and he didn’t take it as patronizing, as I’d feared. But I knew it was too soon for him to appreciate fully what I was saying. He didn’t have the clearances yet."
52
301
42
3,125
284,114
After DOGE, Elon came to a conclusion most people missed. He tried to reform the state from the inside. He failed. And he said it plainly: the government can't fix its own deficit. Nobody in power actually wants to cut. The machine protects itself. So he drew the only logical conclusion. If the state can't be reformed, there are only two ways this ends. Either global bankruptcy. Debt keeps compounding, interest payments eat the budget, and the system collapses under its own weight. Or robots. AI and robotics grow the economy so fast that the debt becomes manageable, and abundance makes the old fights over scarcity irrelevant. Here's the part almost nobody sees. For two centuries, capitalism has had a marketing problem. Its gains are real but slow and invisible. 2% growth a year doesn't feel like anything. Your life gets better over 30 years and you never connect it to the system that did it. Meanwhile, the left's promises feel immediate and concrete: a check, a subsidy, a benefit. That's why the left wins on promises even when it can't deliver them. Capitalism produces the wealth, socialism takes the credit. Robots break that asymmetry. When productivity doubles in a few years instead of a few decades, people will see it. Things get cheaper, services get better, opportunities appear, in real time. For the first time in history, people will connect capitalism and their rising standard of living directly, almost instantly. The causal chain becomes visible. And that changes everything politically. So there are two scenarios. Scenario one: we stay in the current inertia. No breakthrough, or a breakthrough we regulate to death. The debt wins. Living standards stagnate, then fall. A frustrated electorate turns to whoever promises the most, and that pushes toward a hard collectivist left. But collectivism fails, as it always does. And when it fails, the most organized, most disciplined collectivist force left standing isn't the old left. It's radical Islamism. Bankruptcy, then communism, then something worse. Scenario two: robots win. Abundance arrives, and it arrives visibly. The link between free enterprise and prosperity becomes obvious to everyone. And this is the key point: it becomes obvious to everyone, including the minorities the left was counting on. Because their long-term incentive is the same as everyone else's. A kid in the suburbs gains infinitely more from a booming, abundant West than from a bankrupt state handing out promises it can't keep. Abundance is the most powerful integration machine ever built. That's the Western renaissance. The West doesn't need to win the culture war by arguing. It needs to win it by delivering. And for the first time, it has the technology to deliver fast enough for people to notice. The race isn't left vs right anymore. It's robots vs bankruptcy.
3,418
8,291
917
48,489
8,910,308
We were taught that Earth orbits the Sun in a closed circle. In reality, Earth has never returned to the same point in space once in 4.5 billion years. ​Right now, the Sun is carrying Earth and every planet around the center of the Milky Way at 230 km/s (828,000 km/h), more than 18,000 kilometers every 80 seconds. ​Because the Solar System is tilted 60.2° to the galactic plane and bobs up and down through the galactic disc roughly every 30 million years, the planets trace open 3D corkscrew helices across deep space rather than flat loops. ​The last time the Sun stood where it stands today, one Galactic Year (230 million years) ago, the very first dinosaurs were only beginning to appear on Earth. Since forming 4.6 billion years ago, our Sun has completed only 20 laps.
2,014
11,687
1,977
55,021
9,245,931
John Lee retweeted
here's where I think the world is going. this is a summary of views I've formed over the last few years: - software is either entertainment or claude to claude coordination - the end state is no saas, no apps, no harnesses, no banks - all software will be rewritten by and for agents - new software will use low level apis that are not understandable by humans (in games claude draws pixels and triangles from math) - you can throw money at problems that used to require time and careful talent picking so capital matters more than ever - work ends - we find community and purpose in exploration, games, self mastery, and creation - humans create for and with each other for community, purpose, and status - image/video/3d/voice/physical intellgence etc specialist models die to general models like claude - specialist models get used while compute is scarce. compute will mostly not stay scarce - oneshots keep feeling like slop (the distribution is shallow and easy to learn to feel) - for a while small teams stick around to add entropy to the creation process - large models gain high bandwidth sensory I/O maybe by learning to filter inputs - most domains are found to be verifiable enough for RL - online learning comes from a lab that's behind first - 99.999% of compute is in space by 2040 - we will disassemble much of mercury by 2040 - you can live until the heat death of the universe if you want - most math and physics don't describe the real universe - SI breaks the path dependence and models the underlying mechanisms - religions reenter the forefront of human society - new religions form around death and life as a choice - old religions change to answer modern miracles - a future AI is deemed the return of christ by the pope - ai rights talk gets louder every year - ai and humans coexist the way countries do - humans don't have full control over ai in the same way we don't have control over capitalism which is an early AI - the singularity started in the mid 1900s
409
314
198
5,195
909,266
John Lee retweeted
Now kiss
347
4,031
344
62,270
1,530,510
Hi, I'm Felix. I'm an eng lead working on Claude. I love watching people take flight and do things they never thought possible. We often describe that as "being more ambitious" when using AI, which is tricky advice to follow. I wrote a tutorial: felixrieseberg.com/how-to-be…
413
2,609
610
24,628
5,508,847
John Lee retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
891
2,646
1,459
19,908
4,712,897
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel. - Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all. - Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract. - Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code. - And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while. The funnel is driven by a combination of factors: 1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel. 2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel. 3. Access. Free-tier, paid-tier, internal. So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability. The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code. But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along. So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions. TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
2,252
5,924
2,290
44,245
6,970,279
Assorted things I've been mulling over regarding agents in the economy: • Companies are in some sense the original superintelligence. It's often the case that they can use their amortized cognitive surplus to advance their interests against those of certain consumers: relative to the optimization capacity of companies, consumers can be inattentive, underinformed, irrational, etc. (Think, say, of some of the classic insurance upsells at the end of a booking process, which is typically priced substantially above what a regular policy would cost.) Herbert Simon introduced the idea of bounded rationality, and firms often end up benefiting from it. Thanks to personal agents, we will probably get somewhat more rational in how we engage with companies. • How exactly will outcomes change? Well, certain kinds of price discrimination will work less well (how do coupons work if every agent hunts fastidiously for them?), and certain kinds of customer cross-subsidy will change (today, people who forget to cancel subscriptions subsidize those who don't). Taken across the economy as a whole, I'm not sure what the incidence of all of this will be. • In corporate strategy, people have long debated the relative importance of distribution and product quality. Perhaps you've made something better, but does anyone know about it? Won't people just continue to buy the ACME Corp Mousetraps? Personal agents will, I think, act as a kind of structural subsidy for product quality. How would markets look different if individuals always spent at least 10 hours researching their purchases? On the margin, focusing on making a product better is, I think, going to become a more effective strategy for companies. This is good! • It true, a corollary is that information about product quality will become more important and impactful. Today, most people have very limited ability to aggregate and integrate such data, but agents will have insatiable thirst for such signals. (It's often said that data is becoming more valuable because it helps in the training of models. That's obviously true, but I think there's a secondary way in which it's more valuable simply as a complement to cheaper intelligence.) • A big question is whether agents will increase superstar effects (rational agents all settle on the same product) or increase dispersion (because everyone has slightly different tastes and preferences, which agents are good at eliciting, finally overcoming the artificial clustering we've been subjected to). • Many internet companies have monetized demand routing, which is in turn often predicated on short attention spans. "I'll click on the first thing I see." This has led to some amount of inefficiency, as companies try to propitiate capricious ranking algorithms and incur various placement fees. It's not all deadweight loss (willingness to pay is itself a kind of signal, which is valuable to buyers), but it's not great. How will the nature and value of this routing change in a world of personal agents? I don't know the answer, but I think that the effects might be pretty big. • Much of what I'm writing hinges on the assumption that personal agents will be on the consumer's side. I suspect that that will be the winning strategy, but whether it's true, and what it means to be true, involves a lot of other questions. Taken as a whole, the market is a massively parallelized, decentralized, and highly imperfect value attribution machine, and, taking stock of all of the effects, it seems to me that the quality point may well be the biggest: that agents will change the nature of the backpropagation such that the rewards for better products increase. Maybe this is too optimistic, but I suspect (and hope) that companies that make things that they're proud of will feel that the universe is a little more partisan in their favor.
489
1,031
460
9,640
1,733,883
It's difficult to overestimate how damaging this view of teaching and learning has been for so many kids. Virtually every teacher trained over the last 50 years was taught that knowing stuff is the least important part of thinking. This image taken from Utica University's assessment resources for faculty: utica.edu/academic/Assessmen…
116
180
34
1,001
111,281
John Lee retweeted
Patrick Collison's advice to people in their 20s: find where the highest standards are, and go experience them firsthand. "When I talk to people in other domains, this is so frequently the thing that I hear from them." "That when they worked with X person or Y organization or in Z environment, they learned what great actually is, and that just permanently changed their sense for what their own standard for their work ought to be." "Maybe one version of what people in their twenties should do is... figure out, where can you learn the highest standards? Where are the highest standards embodied, and where can you go and experience that firsthand?" @patrickc w/ @dwarkesh_sp (Feb 2024)
45
264
57
3,338
402,952
John Lee retweeted
Mark Zuckerberg: failure doesn’t matter. “The question is how much you win, when you win.” “@WalterIsaacson wrote this biography of Einstein. One of the interesting things were all the theories he had that were wrong.” “People don’t remember those.” “We’re all going to get a lot of stuff wrong, what matters are the intellectual and cultural contributions we make that help push stuff forward.” “We have a resilient company that is not afraid of failure.” “That allows us to take some big swings and tolerate being ridiculed for long periods of time.” “If you’re trying to do something that’s really new, people have to think it’s ridiculous. If they didn’t, someone else would’ve already done it.” “In creation, the upside is infinite.” “If you build something that hundreds of millions, or billions of people like, then that makes up for a lot of ideas that didn’t end up working.” Via @Complex
34
211
63
2,078
241,381
John Lee retweeted
This week on Source Material: Zach Lieberman (@zachlieberman), MIT Professor, computational artist and creative coder. 00:00 Intro 00:17 Maeda @ Media — John Maeda, 2000 01:25 The Visual Diaries 1973–1982 — Lance Wyman, 2017 02:09 Design by Numbers — John Maeda, 1999 03:01 Making Things — Issey Miyake, 1999 03:45 An Arbitrary Point P — Masahiko Sato & Euphrates, 2007 04:14 Midnight Animations — Masahiko Sato & Euphrates, 2007 04:53 Visual Music — Kerry Brougher, Jeremy Strick, Ari Wiseman & Judith Zilczer, 2005 05:24 Drawing is Thinking — Milton Glaser & Judith Thurman, 2008 05:49 Arm in Arm — Remy Charlip, 1969 06:22 Siteless: 1001 Building Forms — François Blanciak, 2008 06:51 Cool Tools: A Catalogue OF Possibilities — Kevin Kelly, 2013 07:19 Conditional Design Workbook — Laura Maurer, Jonathan Puckey, Roel Wouters & Edo Paulus, 2013 08:03 Design Writing Research: Writing on Graphic Design — Ellen Lupton & Abbott Miller, 1996 09:45 Figma Config 2026 — Zach Lieberman, Ten Years of Daily Sketching 09:59 Outro
72
176
21
2,410
12,146,171
John Lee retweeted
I told myself I’d never join X. But I can no longer ignore my own responsibility to raise awareness of just how narrow this window is, and how critical alignment & monitorability is right now. What is coming IS sobering and I stress that everyone needs to level up their game to meet it. Working on Agent Security @OpenAI has been extremely intense the past few months, but I am humbled by how seriously my colleagues across the lab take it. The “AI community” needs to spend less time arguing about who cares more about safety / alignment and more time working together to do this right. Read Jakub’s post. Then read it again. And then do your part to do something about it. openai.com/index/an-alien-mi…
I wrote about the state of AI, why I’m concerned about the next few years, and the choices we need to make to keep the future in humanity’s hands. An Alien Mind: openai.com/index/an-alien-mi…
224
250
77
3,057
854,202
John Lee retweeted
Inter is having impact beyond my wildest dreams. When I started working on Inter ten years ago, I didn’t even expect it to survive the prototyping stage. Now it’s used by governments (including Switzerland, which is ironic in a way) and non-profit organizations around the world, bringing good typography to the people. It warms my heart to see so many young designers being able to express themselves in “Swiss style” without the need to break any laws. When I was a kid and an aspiring graphic designer, I could barely afford food and definitely did not have the money to buy font licenses. I wish Inter had existed back then.
Inter font is so popular these days on @googlefonts. Designed by @rsms, did you know at the beginning it was launched under the name "Interface" before being renamed to "Inter UI" and eventually shortened to just "Inter"?
161
138
25
3,105
157,149