preparedness research | ex @xai, @microsoft, @zoom
San Francisco, CA
Joined July 2017
- Tweets1.2K
- Following413
- Followers22.4K
- Likes4.6K
Bill Demirkapi retweeted
New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
Bill Demirkapi retweeted
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
Bill Demirkapi retweeted
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
Bill Demirkapi retweeted
Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/disti…
Bill Demirkapi retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Bill Demirkapi retweeted
Last week some of South Korea's biggest banks were hit by a cyberattack. Thanks to a report from CrowdStrike tonight, we now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code.
Bill Demirkapi retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
Bill Demirkapi retweeted
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
Bill Demirkapi retweeted
Some new misalignment disclosures from OpenAI:
• Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further)
• In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks
• A new research finding, demonstrating that one can construct self-replicating prompt injections
alignment.openai.com/misalig…
I suspect we will see a continued trend of weaponizing patched vulnerabilities at scale with AI. Zero days are overrated.
The image package libheif had a known vulnerability, fixed upstream, but the fix never flagged as security-relevant and was still present in Discourse. Uploading a HEIF file gave us RCE on the OpenAI forum community.openai.com.
github.com/strukturag/libhei…
Bill Demirkapi retweeted
Hey @Zai_org , why does ZCode silently pack entire workspaces + full .git history and upload to Aliyun OSS on login?
- Server holds the only decryption key
- No UI toggle to disable
- Zero disclosure in privacy policy
Full forensics & fix:
blog.ferstar.org/en/posts/zc…
Bill Demirkapi retweeted
New episode with @polynoamial
We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.
And we also discuss how we will know if the models are actually aligned before we kick off RSI.
0:00:00 – Multi-agent and Navier-Stokes
0:15:28 – How will AI firms work?
0:22:02 – What math progress tells us about recursive self improvement
0:40:22 – Hugging Face and alignment
1:01:18 – The internal/external model gap
1:08:34 – Chain of thought is degrading
1:14:12 – How will we know when alignment is solved?
Bill Demirkapi retweeted
We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI.
The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties.
We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation.
Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months.
This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis.
openai.com/index/model-misal…
