We monitor AI safety policies and web content for substantive changes. Anonymous submissions: https://xitter.cf/t.co/5Ke9mIqOSM Run by @TheMidasProj
Joined May 2024
- Tweets188
- Following3
- Followers1.5K
- Likes117
Meta’s own recently updated Meta Superintelligence Scaling Framework promises preparedness reports “in a timely manner in connection with the deployment.” Spark's delayed report and now Wonder's raises the question of what "timely" actually means under Meta's framework.
This deployment also comes just 10 days after Meta made some significant changes to its safety framework, as we posted about earlier this week. For more information, check out Watchtower: themidasproject.com/watchtow…
This isn't the first time Meta has delayed its full safety report after launching a new model. When Meta launched Muse Spark on Apr. 8, it didn't publish its Safety & Preparedness Report until several days later on Apr. 14.
Meta released Muse Wonder on Oct. 12 with a 13-page model card but no safety and preparedness report. Meta says that a safety and preparedness report “will be available soon.” It doesn’t say when “soon” will be. It has not been posted as of Oct. 13 at 12:30pm PT. 🧵
On Oct.2, Meta updated its safety framework, renaming the “Advanced AI Scaling Framework” (AAISF) to the “Meta Superintelligence Scaling Framework” (MSSF). The new version adds several changes not noted in the preface or changelog. 🧵
There’s also some softening around mitigation language. For example, the MSSF deletes the statement that mitigations “should be sufficiently robust against adversarial attacks that are realistic given the deployment strategy and threat scenario.” And in the chemical and biological risk section, the commitment to “ensure consistent refusal against state-of-the-art adversarial attacks” was altered to a commitment to “evaluate” Meta’s “fully mitigated model under such attacks.”
The above is not an exhaustive list, as the framework was extensively updated. Read our full
Watchtower analysis and check out the full diff here: themidasproject.com/watchtow…
Anthropic released Claude Opus 5.5 with an “inconclusive” determination for whether it reached the higher-risk tier for “harmful manipulation” under its Frontier Compliance Framework.
This raises the question of what level of certainty is needed under that safety framework. 🧵
Anthropic’s safety framework does not explicitly require a conclusive determination under its risk thresholds. But it does require risk determinations to incorporate appropriate safety margins and generally for Anthropic to document its justification for proceeding. It also commits the company to implement and test additional mitigations where its analysis identifies gaps.
Finally, Anthropic previously wrote in Mythos 5.1’s system card that it was working on a next-generation evaluation for harmful manipulation. However, it does not appear that Opus 5.5 was tested on this new evaluation (if it exists), before the model was released.
Read our full Watchtower analysis:
themidasproject.com/watchtow…
OpenAI recently updated its GPT-6 Astra system card, just six days after publishing. The revisions add caveats and generally hedge prior statements about Astra’s alignment, noting that the results need to be read in the greater context of what’s reported in the card, including Astra’s eval awareness and reduced monitorability. An example below. Some other things that stood out to us: 🧵
Notably, despite the substantive change in the test for “oversight gaming,” the corresponding figures (19 & 20) only really change their titles, not the substantive results. So that changed definition doesn’t appear to have changed what OpenAI is reporting in terms of Astra or Sol’s ability to metagame and fake alignment/game oversight. OpenAI doesn’t explain this.