@figbrains

hyperintelligence for humanity

 San Francisco 🌁
fig.inc/
Joined January 2024
  • Tweets103
  • Following5
  • Followers118
Pinned Tweet
fig
@figbrainsSep 29
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly. We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research. Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses. The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
2
5
11
1,454
Join the Fig team tomorrow at Asilomar SSC! Keep up with our research here: info.fig.inc/r/7RGXaZ xitter.cf/Yangyue_Wang/status/21…
locke
@Yangyue_WangOct 26
Headed to the Asilomar Conference on Signals, Systems and Computers tomorrow! I'll be there for @figbrains and @ManifoldRG, presenting GUI-Perturbed. If you're into agents, continual learning, memory, self-improvement, or multi-agent learning, come say hi. I'd love to grab a coffee and talk! Poster TP8a2.24, Tue 1:30pm, Merrill Hall Technical report: fig.inc/blog/domain-randomiz… Paper: arxiv.org/abs/2604.14262 #Asilomar2026

Domain Randomization for Computer Control

GUI models that score 90%+ on benchmarks break under trivial UI changes. We built GUI-Perturbed to measure robustness, not just peak accuracy.

fig.inc
4
144
I'm really excited about the upcoming data technologies the Fig team is developing. We're systematically solving the open data problem. 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝗼𝗻 𝗮𝗽𝗽𝗹𝘆𝗶𝗻𝗴 𝗮𝗶 𝘁𝗼 𝗹𝗼𝗻𝗴 𝗵𝗼𝗿𝗶𝘇𝗼𝗻 𝗼𝗿 𝗼𝗽𝗲𝗻 𝗲𝗻𝗱𝗲𝗱 𝘁𝗮𝘀𝗸𝘀 𝗶𝗻 𝗿𝗲𝗮𝗹𝗶𝘀𝘁𝗶𝗰 𝘀𝗲𝘁𝘁𝗶𝗻𝗴𝘀, 𝗹𝗲𝘁'𝘀 𝗰𝗵𝗮𝘁! Many frontier teams (including us) have run into this big hairy problem of extracting useful signal and building effective, expressive environments for the kind of work someone might do over the course of a full week in a high performance job. We're building a science to solve this. data.fig.inc/preview?utm_sou… xitter.cf/figbrains/status/21136…

Fig · Solving the Open-Ended Data Problem for Models

Data systems that turn real open-ended work in layered, noisy environments into consumable signals to train and configure agents. Request an early preview.

data.fig.inc
fig
@figbrainsOct 23
Real world work is open-ended: multi-objective, interleaved, and noisy. Extracting good learning signals from this has been intractable. We're developing new scalable approaches that turn real open-ended, multi objective knowledge work into signals to train frontier AI systems.
2
3
9
61,167
Real world work is open-ended: multi-objective, interleaved, and noisy. Extracting good learning signals from this has been intractable. We're developing new scalable approaches that turn real open-ended, multi objective knowledge work into signals to train frontier AI systems.
2
1
4
10
62,744
We're sharing early previews with teams ahead of our upcoming releases. If you're deploying models into complex, open-ended environments, we want to work with you: data.fig.inc/preview?utm_sou…

Fig · Solving the Open-Ended Data Problem for Models

Data systems that turn real open-ended work in layered, noisy environments into consumable signals to train and configure agents. Request an early preview.

data.fig.inc
2
67
The Fig team will be in Boston for the ARC Prize summit! Reach out to get a sneak peek at the next generation of intelligence we’ve been quietly building!
harsh✌️
@HarshSikkaOct 22
I’m attending & presenting @arcprize summit in Boston. Will be around tomorrow & Saturday. Dm / reply to me the most cracked people you know, i want to brain drain everyone (researchers, engineers, founders investors anyone) I’ll let them brain drain me too, so it’ll be worth it
2
2
86
Benchmarks hide serious gaps in model performance. Last Friday, we learned the extent of this in "How Good Are Frontier Models, Really?" @Yangyue_Wang from @figbrains dug into why model capabilities remain so jagged even as benchmark scores keep climbing. Missed it? No problem! Watch the full talk here: youtube.com/watch?v=59WZlXqH…

Frontiers: How good are frontier models, really?

Welcome to Frontiers, a series where we bring researchers, engineer...

youtube.com
1
7
1
9
294
A study from the team showing Astra's and Opus 5.5's jagged reasoning behaviors in simple puzzles. More data points on frontier models' failure profiles. xitter.cf/figbrains/status/21100…
fig
@figbrainsOct 13
A few weeks ago, we reported that frontier models make less progress as simple 2D mazes get longer. We tested GPT-6 Astra and Claude Opus 5.5 - length doesn't matter, but they struggle with reasoning about simple obstacles without being told.
1
2
146
fig retweeted
dip@dippogriffOct 13
Cranking up reasoning doesn't solve all problems. More thinking more may generate better text or artifacts, not necessarily actions.
fig
@figbrainsOct 13
A few weeks ago, we reported that frontier models make less progress as simple 2D mazes get longer. We tested GPT-6 Astra and Claude Opus 5.5 - length doesn't matter, but they struggle with reasoning about simple obstacles without being told.
1
3
110
A few weeks ago, we reported that frontier models make less progress as simple 2D mazes get longer. We tested GPT-6 Astra and Claude Opus 5.5 - length doesn't matter, but they struggle with reasoning about simple obstacles without being told.
1
5
4
9
11,352
more replies
That is the gap we care most about at Fig: systems that improve from the consequences of their own actions. Next we put the same mazes in a 3D world and in plain language, with full history, to separate familiarity with the interface from understanding of the task.
1
2
21
Built with @ManifoldRG by Pranav Guruprasad, Sean Rivera, Helen Lu, Arushi Jain and Harsh Sikka.
2
23
Frontier model performance is not uniform across tasks at all, despite what benchmarks say. On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks. Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses. Join us live at 12 PM PT! Registration below ↓
1
2
4
6
392
We've been spending a lot of time thinking about open-ended work and what's needed to capture reliable training signals from it. You can read our latest thoughts here: xitter.cf/figbrains/status/21056…
fig
@figbrainsOct 1

Open-Ended Data is a Bottleneck for AI systems Tackling Real Knowledge Work

Article · @figbrains
1
7
186
Computer-use data focuses on contained tasks in a controlled workspace. Real work is very different: open-ended goals, multi-thousand-step workflows, changing context, and interwoven tasks interrupted and resumed. We wrote about what that gap looks like and why it matters: xitter.cf/figbrains/status/21056…
fig
@figbrainsOct 1

Open-Ended Data is a Bottleneck for AI systems Tackling Real Knowledge Work

Article · @figbrains
1
2
8
549
We expect agents to operate reliably in open-ended worlds. To do so, their training data has to fundamentally evolve: real work observed in continuously evolving environments across unscripted, multi-thousand-step workflows. We’re actively closing that gap. If you’re building models, evals, or benchmarks, let’s collaborate: info.fig.inc/data?utm_source…

Data Requests

Talk to the Fig team about training data, agents, evals, environments, and benchmarks.

info.fig.inc
2
53
Load more