In-context learning for robotics @ skild ai.

Joined October 2022
Samuel Pfrommer retweeted
Humanoid football had a big year. For comparison, this is what the world finals of RoboCup 2026 looked like just last month in South Korea. Should we compete next year? ;) xitter.cf/SkildAI/status/2102807…
We trained a robot to play football. How? Self-play for 140 years in a virtual World Cup. Meet the #Messinator:
17
77
3
339
73,408
The robocup soccer competition has a goal of fielding an all-robot, world-cup-winning team by 2050. Still a ways to go (but we're getting there!)
We trained a robot to play football. How? Self-play for 140 years in a virtual World Cup. Meet the #Messinator:
4
104
Deployment exposes lots of nasty edge cases, this is a great milestone
We just hit 100M ARR within 10 months of starting deployments. We are in factory lines. On construction sites. In kitchens. In data centers. Cleaning. Welding. Building. Cooking. Deploying.
6
217
Samuel Pfrommer retweeted
If S1 makes a mistake, it tries again. For example, the input video prompt here attached the wheel just once. During deployment the wheel didn't align properly, so S1 re-aligned it. By watching the prompt, S1 understands the goal and improvises to get there:
Replying to @SkildAI
what happens if the robot accidentally drops something, does it panic or retry?
36
98
9
673
240,271
Some context: for tasks in the training data, you could suspect that the model ignores the prompt and guesses what to do from pretraining (this was mentioned for both Skild and Generalist's releases). This is plausible when there's only a few objects in the scene with an easily-inferrable intent. Here we show multiple prompts for a fixed kitchen setup, including an example where the prompt contains unusual behavior which is far outside the pretraining distribution (putting a plate in a toaster).
"Does S1 exhibit physical prompt steerability: different prompts induce distinct behavior in the same environment?" Yes! Watch S1 follow 4 different video prompt recipes in the same kitchen. The first one is something far out of distribution -- "putting a plate in a toaster".
2
21
1,836
Samuel Pfrommer retweeted
One surprising aspect of S1’s in-context learning (ICL) is where it shines most: super long-horizon tasks (10+ min) and scenarios outside the pretraining distribution. For short, simple pick-and-place tasks (5–20 sec), most frontier models can already perform well either zero-shot with no prompt or from just a language prompt. With enough pretraining data, the model often already “knows” what needs to be done. @DrJimFan also highlighted this distinction eloquently here: xitter.cf/DrJimFan/status/209046… Once you move to genuinely unseen scenarios and long-horizon tasks, in-context learning becomes critical. This shows up dramatically in the scaling laws: S1’s ICL advantage over regular VLAs grows exponentially as we move further out of distribution. We discuss this in detail in the S1 blog: skild.ai/blogs/s1
Introducing S1, our new foundation model that learns from one example. It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning. Watch S1 operate in real-time via in-context learning:
5
30
1
160
17,093
Samuel Pfrommer retweeted
Feels nice when folks actually read a long blog carefully. Makes countless hours spent with the team obsessing over every single word totally worth it! This is something I learned the hard way over the years from my dear advisor -- Alyosha Efros. Those who know him, know. Haha.
What a crazy statement. Makes me wonder whether Dyna did much data filtering for their model to reach 1M egocentric hours.
3
6
137
14,876
Samuel Pfrommer retweeted
Skild, Generalist, and Sunday published their charts inside five weeks, allowing us to finally measure the progress of robotics foundation models objectively. GPT-3 mattered because in-context learning showed up: put examples in the prompt, get tasks the model never trained on. Sunday's ACT-2 folds laundry at 99.1% in homes it's never seen. But it's one task the lab picked, and the curve is squeezing against its ceiling. It proves scaling perfects for only a chosen skill. Generalist's GEN-1.5: loss falling through 8 months of training, and 59% one-shot success on 10 novel tasks. These results are impressive. But the loss chart tracks prediction error, not task success. And 59% is a single measurement at a single scale. One point can't show you whether scaling is working. Skild tests their robotics model on the most accurate benchmark. In-context learning claims only mean something relative to how far the eval sits from training. They put that distance on the exam, and scale on the x-axis. By that benchmark, the Skild S1 stands alone. On unseen tasks, one video prompt, zero fine-tuning, they are at ~0% at 1k hours. 7% at 10k. 25% at 30k. 66% at 100k. Convex, still bending up. In-context learning should be graded against task unseen and is it long horizon. S1 clears both, using ten minute tasks with dozens of steps that never appear in training. Skild will be sharing more about how they achieved these results soon.👀
Introducing S1, our new foundation model that learns from one example. It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning. Watch S1 operate in real-time via in-context learning:
5
29
4
157
26,996
Samuel Pfrommer retweeted
We’re scaling S1 as fast as we can. The curve shows no signs of slowing down, steadily going up. We’ve been working for a while to find this kind of scaling law in robotics.
Scaling laws for in-context learning for robotics, perhaps the most exciting result in robotics
5
26
3
131
11,585
Samuel Pfrommer retweeted
This is a good question. If the task was only 5–10s long, it can be difficult to distinguish true physical steerability from generalization. But we’re doing tasks that run for over 10 minutes and not seen during training (OOD). Sustaining performance at such long horizon and novel tasks would be impossible without physical steerability actually happening. Hope this clarifies.
Very exciting! But I still have the same question for @SkildAI as I did for Generalist's GEN-1.5: does S1 exhibit *physical prompt steerability*, where different prompts induce distinct behavior in the *same* environment? I like Figs. 5 & 6 because they start to get at this question. Still, there is a confounding factor (as we go from L1 to L5, the scene could simply be getting more OOD from pretraining, which may have a bigger effect on performance than going OOD from the ICL prompt).
3
4
79
12,966
In-context learning for robotics takes careful data curation to get right. For language models, pre-training naturally contains many prompt / response pairs. This is missing in robotic data (outside of some naturally repetitive tasks). Watching S1 quickly master new skills is the amazing culmination of many months of hard work — I'm proud to have helped bring this to life! Some interesting further details in the blog post: skild.ai/blogs/s1?v=1
Introducing S1, our new foundation model that learns from one example. It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning. Watch S1 operate in real-time via in-context learning:
3
27
1,272
Samuel Pfrommer retweeted
We have acquired Zebra Technologies’ robotics arm (formerly Fetch Robotics). This is what happens when orchestration meets intelligence -- a major step toward fully autonomous warehouses. More robots. More environments. One unified brain.
9
47
30
317
86,798
It's an exciting time to bring general robotics to manufacturing!
Nearly every system today, from energy to chips to food, is bottlenecked by scarce human capital. We are changing that by building AI-powered industries of the future. Check out Skild Brain robustly assembling GPU racks, a highly precise task, live at #NvidiaGTC.
1
5
309
Samuel Pfrommer retweeted
Robotics is a data problem. Today, we’re partnering with @ABBRobotics, @Universal_Robot, and @NVIDIARobotics to deploy the Skild Brain across real-world industries from manufacturing to factory lines. This will help us build the world’s biggest data flywheel for physical AI.
19
88
33
720
259,546
🚀 big milestone for the team 🚀
4
167
Samuel Pfrommer retweeted
Let robots clean in peace 🧺🤖✨
6
42
25
195
170,735
Samuel Pfrommer retweeted
giving our humans (and robots) a break at skildsgiving ‘25
5
21
10
114
25,684
Samuel Pfrommer retweeted
Proud to partner with NVIDIA and Foxconn to automate @NVIDIA's Houston GPU Factory and drive the next era of American manufacturing!
21
55
20
366
75,850
Needs a human baseline
can't catch me now 🏃💨🪚🪚
4
120
Samuel Pfrommer retweeted
2
1
1
61
5,807