software, infra, ml | head of eng @tryconversion | ctf @redpwnctf and @dicegangctf | cs @ucberkeley
Joined September 2018
- Tweets380
- Following299
- Followers385
- Likes3.1K
Pinned Tweet
DefCon CTF is over and it's a bittersweet feeling that CTFs as I knew them are gone forever
I'm very grateful I got to experience pre-AI CTFs and meet so many amazing people along the way. Being able to call teammates (from all over the world) friends is a great privilege
RL has really evened the playing field huh
data and environments are king and meta has a lot of real world ones (users)?
Meta has released Muse Wonder and Muse Wonder Full, achieving Artificial Analysis Intelligence Index v4.3 scores of 52 and 56 at max reasoning effort - each a material gain over Muse Spark 1.3 (max) at 48, with a release gap of only 6 weeks
Muse Wonder is available for users on Meta’s first-party API, while Muse Wonder Full is currently only available for Muse Code users. Meta notes that each model is designed for personal and professional agent use
Congratulations to @AIatMeta, @finkd, and @alexandr_wang on the release!
Key Takeaways:
➤ Both models exhibit a material jump in overall Intelligence Index score. Muse Wonder Full (max) improves on Muse Spark 1.3 (max) across all ten Intelligence Index evaluations, gaining 8 points to 56. The largest gains were seen in agentic evaluations: Terminal-Bench 4.0 (+24 points, 33% to 57%), GDPval-AA v2.1 (+176 Elo, 1681 to 1857), and AA-Briefcase (+148 Elo, 1581 to 1729). Muse Wonder (max) gains 4 Intelligence Index points and lands at 52, with notable improvements vs. Muse Spark 1.3 (max) in Terminal-Bench 4.0 (33% to 52%) and AutomationBench-AA (58% to 68%)
➤ Muse Wonder Full (max) places in the top three on six of the ten Intelligence Index evaluations, and is particularly strong in agentic knowledge work and scientific reasoning. In agentic knowledge work, it narrowly leads on GDP.pdf (33%), ranks #2 on GDPval-AA v2.1 (1857 Elo), and #3 on both AA-Briefcase (1729 Elo) and AutomationBench-AA (71%). In scientific reasoning, it tops the CritPt leaderboard at 33% and ranks #3 on SciCode with 63%. This builds on the relative strengths of Muse Spark 1.3 (max), which also showed particular ability in these two domains
➤ Both models use fewer output tokens than Muse Spark 1.3 (max). Muse Wonder (max) and Muse Wonder Full (max) each use around 54k output tokens per Intelligence Index task, down ~10% from Muse Spark 1.3 (max, 60k). This usage is lower than similarly intelligent models Gemini 4 Argon (high, 62k) and Claude Fable 5.1 (max w/ fallback, 78k), but higher than GPT-6.1 Sol (max, 38k) and GPT-6 Astra (max, 27k). Muse Wonder Full (max) uses 16% fewer input tokens per Intelligence Index task than Muse Wonder (max) (6.44M vs 7.69M), meaning it is both more intelligent and more token-efficient
➤ Cost per Task for Muse Wonder (max) lands higher than most similarly intelligent models. Despite efficient output token use, premium API pricing ($4 per million input tokens, $20 per million output tokens) and high input token use (7.69M per task) push Muse Wonder (max)’s cost to $4.52 per Intelligence Index task. This is lower than Claude Fable 5.1 (max w/ fallback, $7.63), Claude Opus 5.5 (max w/ fallback, $5.98), and Claude Sonnet 5.5 (max w/ fallback, $5.46), but higher than GPT-6 Astra (max, $3.26), Gemini 4 Argon (high, $1.99), GPT-6.1 Sol (max, $0.72), and its predecessor Muse Spark 1.3 (max, $1.60). Per-token pricing is not currently available for Muse Wonder Full, so no Cost per Task figure can be estimated at this stage
➤ Both models gain significantly in Coding Agent performance. Muse Wonder Full achieves 68 on the Artificial Analysis Coding Agents Index at max effort in Muse Code, alongside Claude Code with Sonnet 5.5 (max, 68) and Opus 5.5 (max, 66). Muse Wonder (max) scores 63. These represent material gains over Muse Code with Muse Spark 1.3 (max), which scores 54
Other details for the two new models:
➤ Context windows: 1M tokens each, unchanged from Muse Spark 1.3
➤ Pricing: for Muse Wonder, $4 input / $0.20 cached input / $20 output per 1M tokens. Pricing per token is not yet announced for Muse Wonder Full
➤ Input modalities: text, image, video, and audio
➤ Output modalities: text
➤ Availability: Muse Wonder is available on Meta's first-party API and Muse Code at launch, while Muse Wonder Full is only available to Muse Code users
As more sites block personal agents, I’ve really felt its utility drop off a cliff
It’s almost like all the value was just trust + browser use and as monopolies move to protect their territory seems like blocking might become more common
I did not realize there are over 11,000 starlink satellites
That is just way higher than I would’ve thought
Perspective on AI coding over the past 3 years
No way it can tab complete
It can now tab complete 🤯
No way it can agentically code
It can now agentically code 🤯
...
Concern about the future with AI
...
Now, pretty intense optimism
I wonder what I'll be wrong about next
Data is the only moat
What personal agent platforms show is that adoption is driven by access to data. That's implicitly driven by trust or by just having the data
It's why Muse with the backing of Meta is seeing such explosive adoption. All the data is already there
Personal agents have completely replaced "small software" and make creating personal software so easy. Just asked muse to go through saved recipes and make me a website!
These platforms have consolidated vibe coding platforms and chat for me and it's incredible
Jimmy retweeted
That’s always the problem between TLA+ and implementation. Research and industry has the same problems since research often tests narrow use cases.
meta finally got the having its own app store and plans to eventually follow the app store model of taking a portion of transactions
it wasn't the meta glasses that did this but rather just a personalized ai agent... incredible
LLM architecture research has really reached the point of "it'll probably learn okay-ish so just apply as many systems optimizations as possible"
what a fun time!
I want to know how much these RL runs cost so badly - is it millions or billions
And if we get to a point where environments are generated via ai, isn’t it just a hill climb on all knowledge / the world
Jimmy retweeted
Today, I'm excited to launch @tryconversion Mosaic! Apparently the CEO does the launch video now. Nobody told me this in advance 👇
One question I think about is in a hill climbing battle if someone has a head start they will basically continue to maintain it
Leap frogging and catching up to the frontier in anything seems challenging, how has this played out in history?
keeping up with the frontier everywhere is becoming a full time job because things are moving so fast
ai, security, systems; it really feels like what previously would've been years of innovation is now packed into every month and the challenge now is how to mentally keep up
AX + substrate being open source as really an "isolated background task execution engine" is really exciting
There is quite a lot of deep systems knowledge and thinking which were siloed in companies but now an awesome solution is available -- all open source and public!
🥳 This week we announced Agent Substrate. It's the compute runtime (that can work on Kubernetes) that enables everything you've seen from me. We can suspend and resume so fast that you can do it per tool call.
👉 github.com/agent-substrate/s…
👉 github.com/agent-substrate/e…
even more efficient sparse attention mechanism 🤯
