Can LLMs make music?
Introducing MusicArena, the crowdsourced benchmark for AI music-making.
Give a prompt. Watch models make music. Vote for your favorite.
Every vote helps us figure out which models actually have taste.
Make music for free: musicarena.io
Shii might have to try this
3rd party independent embedded evaluators are a must
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Kevin Yao retweeted
Today, LLMs have been thoroughly evaluated on hard skills like math, coding, and question answering. But what about equally important capabilities like long-term planning, negotiation, cooperation, and strategic reasoning? The next generation of AI models will need to plan, reason, and communicate with both human and AI peers.
Strategy and social games have captured these skills for decades. But canonical games like Catan, Mafia, and Monopoly are deeply embedded in the training data that modern LLMs learn from. Models can often recognize these games and recall familiar strategies, making it difficult to distinguish genuine strategic reasoning from rote memorization.
At Catapult, we're building novel strategic environments where success comes from reasoning from first principles, adapting to unfamiliar incentives, and understanding other players' goals and behaviors, not from recognizing a game the model has seen before.
If the future of AI is collaborating with people, the future of AI evaluation should measure those capabilities too.
To follow the latest model leaderboards, play the games yourself, or explore research collaborations, visit playcatapult.io/.










