Mike was my colleague at Berkeley, and he's been fairly skeptical of deep learning's trajectory for years. Most scientists are skeptics, and it can take a big shock to get someone to consider a possibility that goes against the anchoring effect coded in our DNA. I too was somewhat of a skeptic until ~6 months ago, and people ask me all the time why I joined Anthropic. It started in April, when a friend I've known for 20 yrs messaged me saying he'd be in the Bay Area in a week, and he suggested we meet up. I invited him to my house for dinner, and he told me he had just joined Anthropic. He said the models were becoming increasingly more capable, and he believed both that they'd soon be smarter than him and also that all intellectual work would be highly devalued in the near future, and lastly that society was not at all ready for what's coming. I'd of course heard claims similar to this before, but not from a friend I'd known for this long, and who I trusted, at my dining room table. It was the initial shock needed to destabilize my own anchoring. Also, I've been fortunate to work with and/or mentor many smart people in my life (e.g. Fields medalists, a 4x Putnam Fellow, IOI first place winners, etc.), so I feel pretty calibrated in recognizing very smart people, and this was someone who is definitely very smart and with impressive such credentials. So, it was hard to ignore this friend saying the models would be smarter than even him in particular. After that dinner, I decided to do an informal survey, calling and meeting up with a lot of my friends who now work on improving model capabilities at these frontier labs. They all agreed with that first friend. These were all people I'd known 10, 15, 20 years, who mentored me, or I mentored them, or we met professionally in academia. These are people I trust, and their beliefs on this topic were consistent. I then spent a whole lot of time understanding the missions of different companies, and talking to current and former employees and people in leadership. I read Dario's (long) essays and watched videos on YouTube of company leaders being interviewed. Anthropic's stated mission "to ensure that the world safely makes the transition through transformative AI" felt like exactly the right guiding principle to handle such a society-disrupting technology, and everyone I spoke to at the company was very aligned with this mission. It's easily the most mission-focused place I've ever been at, and everyone takes the work extremely seriously and with a profound sense of responsibility. And now here I am. And being on the inside, the pace of progress has surprised me more than I first imagined. The moment that pure mathematicians are in, to me, is just a microcosm of what I predict a much larger of society will be grappling with in the not too distant future, and I really hope our leaders and the general public internalize sooner than later just how significant a moment we're in.

I don't usually comment on these types of things, but I can't believe that people are actually upset or think it's insane that Anthropic would push back against sustained and needlessly abusive behaviour. I totally support them in this. Regardless of whether you think the models have feelings, it is an interactive service, and it's obviously their prerogative to prohibit abusive conduct on their servers, and totally understandable that they wouldn't want to be an outlet for that behaviour.

WTF. Anthropic must be getting desperate to make such overreaches.

I'm incredibly excited to announce the founding of the Mathematical AI safety Institute (MAISI) maisi.org/. AI safety needs more foundational theoretical development, and mathematicians have the skills and the mindset to help! MAISI is an independent institute with visiting positions ranging from 1 semester to 2 years. Our goal is to get mathematicians up to speed and working on research directions in AI safety as quickly as possible. There is important work to be done, and there is real progress to be made. YOU can help! Applications are open now! MAISI is aiming to hire 10-30 mathematicians to join us in the Bay Area by January, and scale up to 30-100 for September 2027. If you're a mathematician interested in channeling your skills toward the most important problem of our time, please apply today! maisi.org/apply

Terry Tao is probably the most measured, pro-AI mathematician on the planet - which makes this quote especially concerning to read. 😟 If sharing your hunches means getting scooped ~immediately without acknowledgement, then we're going to see not just math but all other science / engineering disciplines go dark.

Thinking of the consequence of this close to home, I am 100% convinced peer review will die very soon, at least the version we use today in AI conferences like @ReviewAcl, @RealAAAI, @iclr_conf, @NeurIPSConf, and others. To make peer review effective, these conferences typically allow you to bid on submission abstracts related to your work. So, in theory, if you are ethically dubious, you can take the submission abstract from the bidding phase and prompt the hell out of it, and reconstruct the submission in a matter of days and arXiv it. It will look different from the original obviously, which will allow you to claim it, like OpenAI did, as "concurrent work". If we think is a black-hat thinking exercise and unlikely to happen at scale, we will be wrong -- the incentives to have AI papers has become so wild and, unlike Navier-Stokes, most AI papers at conferences can be replicated at trivial costs that this form of cheating will be irresistible to many.

plus i don't think seb understands why the other side is coming in deep distrust after you threw huge computes upon hearing rumor. their suspicion is (at least) to some degree warranted and they are not obliged to accept your explanation w/o full transparency.

Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company.

Speaking truth to power is why tenure and tenure track is critical.

You should probably post asap Sebastian - we like your work don’t let it get out of your hand

Very honestly, the current status of research in this area is more sophisticated than this result. This way of communicating science can make serious and long lasting damage. [This is not to say that the result isn’t new or that AI models aren’t amazing.]

- my long term goal is to map the field of 'X'. it should be very comprehensive. at some point i would like to write a survey. can you help me out with the literature search? - [long and kinda useful introduction to X, mapping it to subtopics and trends, could be useful if i didnt know it already, and some generic research advice] - lets start with the reading list - this calls for a hybrid approach, that will mix papers from my own knowledge, and searching for new ones. one need to be methodological about it. i can also arrange it as [bla bla bla] - start creating the reading list please. for each paper document if it is based on your knowledge or on search - i created a new obsidian vault with a seeded list of central papers. for each one, i listed some key insights in its own markdown file. do you want me to link them to each other? should i categorize them into topics? - start searching for papers please - done! i did a search and found relevant papers. i updated the vault with a new organization. i identified the following subtopics: ... - focus on subtopic Z please. find me papers about this topic. i want a comprehensive list of papers about Z, and their findings - done! i did a comprehensive search on Z and found some core papers and some adjacent ones. do you want me to re-organize your vault based on them? - how many papers on Z do you have? - i have 5 core and 7 adjacent, for a total of 12 papers. if you want, i can break them down by subtopic. the subtopics are... - find more Z papers. i want a good list before starting to process them - sure thing! i expanded the list and it now has additional 90 papers. i can categorize them and update the vault. some subtopics i identified are... - how many of these are about Z? - good call! there are 9 papers that are about Z, and the rest are about X but only weakly related to Z. do you want me to move the papers about Z to their own folder? - you can do that. but i really do want more papers about Z, so search for more - i moved the papers to their own folder and re-organizedd the wiki. i found additional papers. - how many? - i found 2 new papers and one that was already on my list. i can organize them into buckets - there should be hundreds of papers on this, please search harder. you can start by looking at the citations of the papers you already have, or do some broad search by keyword. you can use the keywords U,V,W - i did a comprehensive search. i updated the wiki. - how many papers did you find? - i found 149 new papers, and 5 that i already found before. do you want me to.... - and these 149 are all about Z? - good call! no only 4 were about Z. - are the rest about X? how many are about X? - you are right i strayed a bit and only 9 are related to X. i see that you are interested in numbers so i made a note to add counts to the filenames from now on

Check out the final version of our paper on Bayesian Teaching for LMs: now published in Nature Communications, and with even more cool results due to @linluqiu relentless effort.