@SamWolfstone
Account based inUnited Kingdom
About this account
- Account based in
- United Kingdom
- Connected via
- United Kingdom Android App
Account-level information from X, not a live location or the device used for a specific post.
Sculpting, AI, Philosophy, Coding.
London, UK
Joined November 2020
- Tweets877
- Following171
- Followers263
- Likes22.3K
Getting AI agents to create text-based adventure games exposes so many interesting capabilities (or lack thereof).
Opus 5.5 made a really wonderful little game. Right level of challenge, good story, world was filled with the right mix of mundane and interesting details. Good pacing, and the puzzles were actually pretty fun, with one really satisfying ah-hah moment.
So, Opus 5.5 doesn't just have good visual tastes, as we've seen from plenty of demos. It's got decent writing chops and fairly good designer-to-player theory of mind.
Sam Wolfstone retweeted
ApprenticeBench, just like a real job, is a comprehensive test: computer use, continual learning with memory notes, long-horizon... if a model has any weaknesses in any aspect, it shows.
> strong differentiation across models
> frontier models are already more accurate than human professionals, but also come with a 250% cost
> large gap between open (Kimi K3's 18%) and closed (Fable 5.1's 72%). Open models are not necessarily cheaper either, because they may spend a lot of tokens when struggling.
Sam Wolfstone retweeted
I think Astra follows the prompt much more literally than Fable. this can be good or bad. if the prompt is garbage, Astra will implement thrash, while Fable will average it out into a more sensible interpretation. if it is great, Astra will do what you want, while Fable will try to outsmart you and ship something worse
I wonder how difficult it would be to RL a model to use a certain special tag at the end of hidden reasoning chains which should be preserved between turns. For a game-playing agent, currently I'm stuck between abandoning reasoning between turns (and thus forget plans that only appeared in the reasoning) and keeping all reasoning between turns.
Anyone else notice Ox Alpha falling into weird loops of using the same incorrect tool over and over hundreds of times? Trying to figure out the situations that trigger the behaviour...
Spent a few weeks creating a harness that allows an LLM agent to play an arbitrary old-school MUD, with the hope that I can then create a whole AI player-base and community/society in a custom text-based world, and visit it myself. That's long-term, though. Short-term goal is just using custom MUD worlds as evals for models, and playing alongside my Hermes agent in some of my childhood MUDs if the admins are okay with something that's technically a 'bot' playing their game.
Given these models' eagerness to communicate and collaborate, I wonder if their release will lead to a resurgence of Moltbook and similar platforms.
I think the only thing standing between current AI and AGI is what I call 'wisdom' (what some call judgement/taste): the ability to make use of the knowledge/experience you already have (or your awareness of a lack of knowledge), to do things more successfully, without having to be told which knowledge to use explicitly.
I've observed LLMs getting better at this over time. But every current failure mode I encounter, everything that makes me think I can't trust an LLM to do everything for me, is some version of that.
OpenAI facing criminal penalties for the felonies committed by its models would be good, in that it would set the correct precedent for future incidents.
Sam Wolfstone retweeted
"Total LessWrong Victory, in the sense that everything is going as predicted, and also a Total LessWrong Defeat, in the sense that everything is going as predicted."
Sam Wolfstone retweeted
A junior dev asked the staff engineer: "How do I know if the AI's code is correct?"
The staff engineer replied: "How do you know if yours is?"
The junior dev was enlightened. Prod went down.
Fable 5 storing its own decisions as the user's decisions in its memory, reported by a much better dev than me. Wondering how much this is a 'Fable is overconfident' issue and how much of it is the usual 'user-assistant-role confusion' we often see. @claudeai
Even if Fable 6 gets restricted to US nationals only or trusted partners, once Fable 7 is available to them instead and they've used it for further patches/defence, Fable 6 would then be made available to the public/globally, right?
Sam Wolfstone retweeted
The best way to understand a complex system is via edge cases and failure modes, because they define the contour of the system.
Sam Wolfstone retweeted
It's a tricky one - Science fiction stories are almost never what the future will look like. But the future will for sure look like a science fiction story.
A large majority of Anthropic/OpenAI paying customers are outside the US. I wonder if they'll be able to make larger models like Fable 5 profitable if those end up restricted to US nationals?