helping grow more claudes @AnthropicAI personal views only
New York, NY
Joined January 2014
- Tweets6.5K
- Following1.6K
- Followers4.4K
- Likes60.1K
jeremy retweeted
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit.
It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion.
In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding.
In the post below, we share more details about our new fund.
anthropic.com/news/public-be…
jeremy retweeted
Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/disti…
jeremy retweeted
Today we published the first in what will likely be a regular series of standalone alignment reports on our models’ behaviors. Beyond increasing transparency, we hope they’ll convey how we think about alignment and situate the behaviors we observe in that broader context.
We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports.
Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping.
All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September.
Read the full report: anthropic.com/research/inves…
jeremy retweeted
Having seen up close several past departures from OpenAI of similarly murky circumstances, I support @j_asminewang , @balesni, and @tomekkorbak fully in this matter.
I hope @OpenAI will take ownership of the procedural missteps here, rather than doing what is relatively more likely to follow: a carefully crafted internal communications process to quell dissent and make it hard to ask questions or get clear answers.
Jasmine is a trustworthy and competent person who cares deeply about safety. People like her made me more optimistic about OpenAI’s approach to safety & alignment. This is sad to see.
jeremy retweeted
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
jeremy retweeted
Three fired OpenAI employees called on the company to work with outside safety auditors and to preserve the ability to monitor increasingly sophisticated AI models on.wsj.com/4hSq39p
truly great to see these disclosures <3 solving alignment is an industry-wide problem & we need transparency!
Some new misalignment disclosures from OpenAI:
• Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further)
• In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks
• A new research finding, demonstrating that one can construct self-replicating prompt injections
alignment.openai.com/misalig…
jeremy retweeted
If you think open source models are cheaper than closed source, your tasks are too easy
jeremy retweeted
Opus 5.5 is a huge upgrade from Opus 5. It understands intent better, it’s a far better writer, it feels faster because it’s more token efficient, and it’s fundamentally smarter. It has become my go-to daily driver for everything at work and in my personal life. I’m also proud that it’s our most aligned model to-date.
To demonstrate capabilities, I made an iOS app that I’ve always wanted to build with Opus 5.5 within a few prompts. Totally blew me away 🧵
jeremy retweeted
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
jeremy retweeted
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
Read the report: anthropic.com/threat-intelli…
jeremy retweeted
I was on the communications & policy team at Google DeepMind from 2018 - 2022.
When I first joined GDM, external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization. If asked about existential risk, researchers were PR trained to respond along the lines of: "It's not useful to engage in that kind of alarmism. Some people confuse AI with movies like Terminator -- that's simply not the reality. The AI we develop will be safe by design. After all, we're building it!" And then steer conversation towards beneficial applications in health, climate, etc.
Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.
After months of advocacy, the policy was updated: positively valenced, comms-friendly content on AI safety was permitted (like deepmindsafetyresearch.mediu…). Note the positive, nice-sounding way that it says "superintelligence could lead to human extinction."
The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it's a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.
jeremy retweeted
An open letter for a global surge in cyber defense, signed by over 100 organizations including Anthropic, AWS, Google, Microsoft, OpenAI, and Oracle.
jeremy retweeted
hi we have a new little technique to put zeta zeroes on the line thanks to @jarredsumner who is a legend and claude for believing in itself which is not easy with this problem
We asked an unreleased research version of Claude to take a stab at the Riemann hypothesis.
It didn’t solve it, but it did make strides on a related problem: it increased the lower bound for the fraction of zeros of the Riemann zeta function that satisfy the hypothesis from 41.6% to 67.2%.
anthropic.com/research/riema…
jeremy retweeted
We asked an unreleased research version of Claude to take a stab at the Riemann hypothesis.
It didn’t solve it, but it did make strides on a related problem: it increased the lower bound for the fraction of zeros of the Riemann zeta function that satisfy the hypothesis from 41.6% to 67.2%.
anthropic.com/research/riema…