Working on https://xitter.cf/t.co/ur0WT8wp70 | All my issues are skill issues

Joined August 2024
Can't see the SVG? Fill issue
Eaten by a whale? Krill issue
1
44
Jonathan Hefner retweeted
In 7 months, the skills.⁠sh registry reached 1 million agent skills and nearly 280 million installs. See what people are teaching agents, and where skills are going ↓ vercel.com/blog/state-of-age…
60
9
21
167
92,560
In a future filled with agent-driven dependency upgrades, semantic versioning has less value. (Ironically, agents are probably better at enforcing strict semantic versioning than humans are.)
2
106
So pretty...🤩
We built vgpu.sh to ship shaders on vercel.com. Now it's open source. ▪︎ Minimal agent-first WebGPU library ▪︎ Run in the browser or headless Node.js ▪︎ Render in CPU sandboxes and CI tests ▪︎ Create reusable .wgsl modules
7
243
Completing my first month at Vercel today. Extremely proud to be part of this ship! ▲
12
1,074
Jonathan Hefner retweeted
thinking of doing skillsbench 2.0. below are some of my personal ideas and would appreciate any feedback 🙌 topics we are thinking of covering: - meta skills/tasks: agents self learning from environments / episodes (skillopt, skill self play) - memory: skill as a media of memory for long running agents - enterprise: skills are used more as a bespoke software at companies + becoming a way of distributing developer experiences: salesforce cli, gws cli (dead tho), hubspost cli etc. previously we have built out mock environments (clawsbench) thus we are thinking of creating sandboxes to test these skills - more capability axes: covering using browsers (since we have mock envs), multimodal, deep search (general reasoning released back search) - elo based: so the benchmark itself is longer lived
11
1
1
34
2,232
Jonathan Hefner retweeted
if you're not using skills you are wasting your time repeatability has always been the goal of software efficiency, and skills are a version of that even before skills were a thing, anyone who was getting shit done had some home grown version of it (embedded docs, macros, etc)
I don’t use loops, skills, fancy orchestration, plugins or anything like that. I raw dog the agent with its default harness, I read the code, and I’m shipping more than ever to production for real projects used by real people. Sometimes if it’s a low risk part of the application I might skip reading the code but only sometimes. I think most are either lying about the amount of value they are creating or just having fun while understanding they aren’t creating value but failing to distinguish that to their audience.
31
6
4
345
31,159
I agree more and more. Everything seems to point to doing work in passes -- code generation pass, review pass, etc. "Dreaming" is yet another pass. I think we'd greatly benefit from more RL on memory extraction and/or discovery.
Sleep-time compute is the next scaling axis for intelligence
1
2
1
5
1,666
Jonathan Hefner retweeted
A big pain point in using AI benchmarks is encountering errors after its first release. Today, we're releasing SkillsBench 1.1, the first benchmark for how well AI agents use skills, now audited end to end and verified error-free. Prof. @dawnsongtweets joins 1.1 as advising author. We worked through every task with several frontier labs to eliminate the errors in the previous version. We also added new tasks, moved the ones with external dependencies into a separate set so the core suite runs clean, and expanded coverage to more models. Capability is climbing fast. The best with-skills resolution rate rose from ~36% (Claude Sonnet 4.5, Sep 2025) to 67% (GPT-5.5, May 2026), about +1.9 points per month. The frontier is hill-climbing SkillsBench fast. The right skills still matter. Across the fleet, curated skills lift resolution rate by +16.6 points on average (33.9% → 50.5%), and by as much as +25.7 points for a single model. The top configuration is GPT-5.5 on OpenHands at 67.3%. By popular demand (thx Nate @cursor_ai), we're now tracking skills invocation: how often an agent actually uses the skills it's given. Recent flagship configurations invoke them 90–99% of the time (Codex 99%, OpenHands + GPT-5.5 92%, Gemini CLI 90%), versus roughly 50% for older setups. Also new in 1.1: @OpenHands joins as a fourth harness, alongside Claude Code, Codex, and Gemini CLI; a rebuilt leaderboard with refined categories, subdomain skill rankings, and Skill Lift; and native task . md on BenchFlow, with multi-scene environments and rollout branching. We also partnered with @k_dense_ai to add scientific skills to some science tasks. One implication for deployment: skills can substitute for scale. GLM 5.1 with skills (58.4%) outperforms Opus 4.8 without (45.7%). A smaller model with the right procedural knowledge can beat a larger one running without it. Huge thanks to @nick_kango @ivanleomk @kaggle @GoogleDeepMind for hosting a launch event with us. Thanks for everyone who's come on May 27! Also thanks to our partners @gneubig @OpenHandsDev @ivanburazin @daytonaio @jackminong @johannes_hage @PrimeIntellect @TimothyKassis @k_dense_ai for providing support in credits, compute, and skills. SkillsBench live leaderboard will also come to @ValsAI. Many people have told us they use SkillsBench as an index to measure models' agentic capability over diverse and high GDP value domains. Great work on Valkyrie as well! @ Jarett @nikilravi @langstonnashold @RayanKrishnan SkillsBench is fully open-source. Explore the leaderboard and tasks, read the docs, or contribute your own skill set or harness and join the leaderboard. 🧵
12
29
9
127
19,253
How much of taste in code is visual? Like having a visceral reaction when you see nested ternary expressions or a pyramid of doom. How much taste in code could models gain by having better vision and seeing rendered code during review?
78
In addition to leaking neuralese, models can have bothersome verbal or behavioral tics. I can imagine a future where users only interact with "chief of staff" agents that embody and enforce user preferences, delegating all work to other agents and acting as a filter.
Replying to @tamaybes
fascinating because 5.5 does this too. invent weird technical jargon. perhaps you’re right and it’s a neuralese leakage
3
95
The state of agentic coding right now:
1
65
Jonathan Hefner retweeted
Grok now has skills
Skills in Grok Web can be used by typing /
4,234
4,136
313
46,096
23,590,006
Some people argue that skills will become obsolete as models improve. That may be true to some extent, but models will never know exactly how you (or your company) want things done. Personalization is a fantastic (and often overlooked) use case for skills.
we made a skill to help coding agents + humans write & design in the vercel way - how to write and use our design system - consistent terminology and workflows - clear & concise writing
2
156
Jonathan Hefner retweeted
I’ve seen people on X dunking on folks like @garrytan @doodlestein and others for sharing SKILL dot md files they've built. They are dismissing these files as "just a markdown file.” I think this misses the point entirely and I'll try to address that here. Quick thread: A bad skill file is just text, sure. A good skill file is compressed expertise, packaged in a format an agent can actually use. The value is not just in the “markdown file.” The value is the interaction between: a huge neural network with latent capabilities a precise, reusable, agent-readable procedure that steers those capabilities toward a specific outcome That combination is the product. Saying “it’s just markdown” is like saying Hamlet is “just ink on paper,” or Einstein’s relativity paper was “just a text.” Technically true. Intellectually useless. The medium is simple. The content is what matters. And more importantly, the effect of that content on the reader is what matters. With humans, a book, a coach, a lecture, or painting can change how someone thinks and acts. With LLMs, text is also the control surface. These models were trained on text, reason through text, call tools through text, and follow procedures through text. So yes, the skill is “just text.” But it is text designed to be read by an enormous neural net. That matters. A good skill is agent-ergonomic. It does not merely say “do this better.” It encodes workflow, constraints, examples, edge cases, tool usage, failure modes, and success criteria in a way the agent can reliably execute. That is very different from a casual prompt. A prompt is often a one-off request. A skill can be reused, versioned, tested, improved, shared, and loaded at the exact moment an agent needs it. That turns “vibes-based prompting” into something closer to operational knowledge. Another way to think about it: We have built these massive models, but much of their power is latent. Different people can extract very different levels of performance from the same model. A good skill is a way to actualize a specific slice of that latent capability. A refactoring skill. A research skill. A legal review skill. A math explanation skill. A codebase-navigation skill. Each one can make the same model behave very differently. I think of Cus D’Amato and Mike Tyson. Tyson had enormous latent potential. But Cus gave him a system, a style, a discipline, a way to channel that potential. That’s what good skills are for agents. They are not magic. They are not all equally valuable. Many will be mediocre or useless. But dismissing them right off the batt because they are “just markdown” shows a misunderstanding of what LLMs are. Text is how we trained these systems. (for the most part) Text is how we steer them. Text is how we unlock parts of what they can do. The question is not whether a skill file is “just text.” The question is whether the text reliably makes the model perform better at a valuable task. If yes, then it is not “just markdown.” It is leverage.
10
7
2
50
11,741
People tend to discount momentum. When a company makes a bad choice, people predict failure, but momentum can paper over bad choices. And it cuts both ways: companies assume success is due to correct choices instead of momentum — when momentum eventually runs out, they fall hard.
95
Using probabilistically-weighted average of score output tokens to improve LLM-as-a-Judge effectiveness with higher score granularities. Freaking clever.
Strongly recommend the LLM-as-a-Verifier writeup. Biggest takeaway for me is that increasing scoring granularity makes the verifier more effective. This indicates that LLM judges / verifiers are developing new (and better) capabilities. This did not work well 1-2 years ago. In fact, LLM-as-a-Judge best practice was that lower scoring granularity (e.g., binary, ternary, or 1-5 Likert score) worked way better than granular scores (e.g., 1-100 scale). This was a constant recommendation I gave for setting up LLM judges properly. It seems like recent frontier LLMs now are better at scoring at finer granularities, making this best practice (potentially) obsolete. One caveat to this finding is that the scoring setup used in this writeup is a specific setup based upon logprobs. Instead of just using the score token outputted by the LLM as the result, they compute the logprob of each possible score token and take a weighted average of scores (with weights given by probabilities). Then, they go further by expanding this weighted average across repeated verifications and multiple criterion: Reward = (1 / CK) * ∑_{c=1}^{C} ∑_{k=1}^{K} ∑_{g=1}^{G} score_logprob * score_value where C is the total number of evaluation criterion, K is the number of repeated verifications, and G is the scoring granularity (i.e., number of unique scoring output options). The reward determines if a particular output passes verification across criteria. When using this logprob setup, we see consistent gains in verifier accuracy by: - Increasing scoring granularity G. - Increasing repeated verifications K. - Increasing the number of evaluation criterion C. The last two findings are in line with prior work, but the fact that higher scoring granularity is helpful is interesting! In the LLM-as-a-Verifier paper, this system is used at inference time in a pairwise fashion as described below. "To pick the best trajectory among N candidates for a given task, a round-robin tournament is conducted. For every pair (i, j) the verifier produces Reward(i) and Reward(j) using the formula above. The trajectory with the higher reward receives a win, and the trajectory with the most wins across all \binom{N}{2} pairs is selected."
1
119