70k ⭐️ OSS @llamafactory_ai, CTO @prismshadow_ai, ex-@ByteDanceSeed_ Building PenguinHarness - Best Self-Improving Harness. Opinions are my own.
Joined July 2024
- Tweets477
- Following613
- Followers1.4K
- Likes1.3K
Pinned Tweet
👋 Hi everyone! Long time no see
We're the LlamaFactory team. After passing 70k stars on GitHub, we're now building the "LlamaFactory" of recursive self-improvement (RSI), towards a unified, efficient, stable, and flexible framework for agent self-evolution. Excited to share our team's latest work with you all.
TL;DR:
🐧 We've open-sourced PenguinHarness, a multi-agent self-evolution framework that brings every RSI algorithm under one roof, over 2,600 GitHub stars.
📖 To get a fuller picture of existing RSI work, we went through 140 papers, organized and categorized them into the awesome-rsi repo, and published a survey paper.
📊 To compare self-evolution efficiency across algorithms fairly, we built GDPevo, a self-evolution benchmark covering 6 enterprise scenarios and 240 tasks, and used it to evaluate 6 mainstream agents.
The details:
Following the same philosophy as LlamaFactory, we want to bring all RSI-related data, models, algorithms, and benchmarks into a unified open-source harness, so everyone can try and debug RSI more easily. We've already integrated 5 RSI algorithms and 7 benchmarks, and we're aiming for 100+ down the road.
Borrowing from DSH's "everything is a plugin" approach, we've packaged every algorithm and benchmark as a plugin. They're fully decoupled, so users can freely mix and match any RSI algorithm to evolve different agents. We also built a modern desktop client that installs in one click on any OS and runs locally, so you can grab the latest version from our website.
Website: penguin.ooo/
OSS Repo: github.com/Prism-Shadow/peng…
To better understand where the academic work stands today, we open-sourced the awesome-rsi repo and published a survey paper covering 140 recent works. We also mapped the trajectory of RSI against that of deep learning — many concepts line up one-to-one, only now the subject is an agent.
OSS Repo: github.com/Prism-Shadow/awes…
Paper: arxiv.org/abs/2610.14625
To better evaluate how different harnesses and methods perform at self-evolution, we built the GDPevo benchmark, spanning 6 real-world enterprise scenarios and 240 tasks. With a train/val split, it fairly measures an agent's self-evolution ability while guarding against overfitting.
OSS Repo: github.com/Prism-Shadow/GDPe…
Paper: arxiv.org/abs/2608.03764
This post isn't just about our work. We're also hoping to invite friends from academia and industry to join our open-source project. If you'd like to see your name on an open-source project and co-author papers with us, DM us and let's talk!
Yaowei Zheng | LlamaFactory retweeted
New Blog: Towards the Agentic Era of Video Understanding
For complex tasks, agents have become the default. Nobody uses chat mode for that work anymore: people hand it to Claude Code, Codex or Antigravity (yes sometimes🤣), which open files, run commands and check their own results. We think video understanding should benefit from this too.
So we are releasing Video-Index 1.1 with an agentic evaluation setup (which should be by default!). more in the threads🧵
Yaowei Zheng | LlamaFactory retweeted
🚀 DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.
🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
Yaowei Zheng | LlamaFactory retweeted
这是我正在做的 DSH-LayerFS。
github.com/Ephemeral-AI-Lab/…
基于 LayerFS 的存算隔离架构和存储压缩能力，做了一个极简的 multi-agent harness：
- Agent 的每一步执行都有记录可追溯
- 每条 Bash 命令都在独立的 Workspace 中运行
- 内置冲突检测机制
- 支持递归嵌套 Fork（Recursive Nested Forking）
再配上这么可爱的鲨鱼娘视图——
大家愿意再给 DSH 一次机会吗？🦈
@tianyi
Yaowei Zheng | LlamaFactory retweeted
Autoresearch with coding agents can now run for days on a single problem, but we've seen an increasing number of cases where they stop improving early. We address this issue with a simple idea: forgetting.
Introducing our new paper: "Scaling Discovery Requires Forgetting"
Yaowei Zheng | LlamaFactory retweeted
还是来正式宣传一下，感觉今天 gosim 有点虎头蛇尾（悲
我们做了 obelisk 这个项目，作为提供给 coding agent 检索自己 session 能力的一个 infra，同时也让人可以很好的审计 coding agent session
仓库：github.com/tommy0103/obelisk
Q1：为什么市面上有那么多 memory 产品，我们仍然需要一个这样的 infra？
A1：
1. 因为很多 memory 都是黑盒产品，raw session 仅在提取 memory item 时，在后续需要用到这些提取产物时，agent 没有回查原始 session 的意愿和能力，它甚至不知道被注入上下文的 memory 的来源是什么
2. 因为被提取的 memory item 是有其 scope 的，而这些 scope 并不被 agent 使用时所知，agent 会扩大这些结论，从而失去准确性；并且我们也不应该认为被提取出来的内容能够真的对现在有效，因为现在需要什么是过去所无法知道的
3. 这样的一个 infra 使得使用 history 成为一个完全透明的过程（对人和 agent 都是），代价是 search 的成本被转移到了 agent 的上下文窗口里，但我们可以使用 subagent 来解决这个问题；它本质就是为 agentic research 提供了材料——历史 session 内容
4.我们做了一些 performance 的测量，在一些 benchmark 上的表现是优于 mem0 的，这部分可以说明它不劣于传统 memory；我们还在寻找更有力的证明
Q2：你们是如何构建 session 检索层的？
A2：
这个问题说简单也不简单，底层是普通数据库选型，没有任何魔法——对，我们放弃了向量匹配，也就不需要向量数据库
我们之前在一篇文章里详细阐述过为什么我们只需要 bm25 / fts5 这种精确匹配，可以查看 obelisk.antinomie.org/blog/，简单来说一方面就是向量匹配有模糊性，而我们不想要模糊性，另一方面是，在 agentic search 中 one shot 的召回率并没有那么重要，但是准确率很重要，召回错误的 item 比不召回任何东西的 negative effect 影响更大
并且我们使用了 codemode 作为面向 agent 的 interface，这使得 agent 可以很好组合各项查询，以及控制上下文长度，这里面有一些 trick
Q3：我想知道 agent 在这个 infra 上的 performance 如何？它是否只能处理一些简单的任务？比如这个代码之前为什么这样写？
A3:
目前我们试过一些非常有挑战性的任务，这其实完全取决于 agent + model 能力，越强表现越好；我们尝试过简单的 memory 任务，让 agent 看看这个项目的代码为什么这么写，让 agent 获取另一个 task 的进度，或者做 handoff，也试过难一些的，让 agent 写周报，或者画出项目 roadmap（搭配 Claude Artifact 效果更佳），甚至更长期的，对于项目选型，架构，coding 做长达一两个月的复盘（请见上面的文章内容）
这些产物都不是简单的 memory item 能够涵盖的，我相信这是一个 aha moment
如果还有其他 qa 也欢迎~
大家好！很高兴分享我们团队的开源新作 PenguinHarness～
在 Github 获得了 7 万 Star 以后，我们正在打造自进化（RSI）领域的 LlamaFactory，志在成为统一、高效、稳定、灵活的 Agent 自进化框架。
一句话介绍：
🐧 我们开源了 Multi-Agent 自进化框架：PenguinHarness，将所有 RSI 算法集成到一个统一框架中，现已获得了 2600 Github Stars。
📖 为了更全面地了解现有 RSI 工作，我们详细调研了 140 篇论文，将它们归纳整理分类，形成了 awesome-rsi 仓库，并发表了综述 paper。
📊 为了公平对比不同算法的自进化效率，我们构建了自进化 Benchmark：GDPevo。涵盖 6 大企业场景、240 个 Task，对 6 种主流 Agent 进行了评测。
以下为详细介绍：
和 LlamaFactory 的思路一样，我们想把所有和 RSI 有关的数据、模型、算法和评估都集成到一个开源 Harness 框架里面，方便大家实验和调试。现在已经集成了 5 种 RSI 算法和 7 个 Benchmark，未来我们的目标是扩充到 100 多个。同时，我们借鉴 DSH 的 Everything is Plugin 思路，把所有的算法和 Benchmark 都做成插件，相互解耦，让用户可以灵活调用任意 RSI 算法进化不同的 Agent。我们也做了精美的客户端，任何操作系统都可以一键安装，本地运行，可以到官网下载最新版体验。
项目官网：penguin.ooo/
开源仓库：github.com/Prism-Shadow/peng…
为了对现有学界工作有更好的理解，我们开源了 awesome-rsi 仓库，并且发表了综述论文，对近期的 140 篇工作进行了梳理，还把 RSI 的发展路线和深度学习的路线进行了对照，发现很多概念都是一一对应的，只不过现在的对象变成了 Agent。
开源仓库：github.com/Prism-Shadow/awes…
论文链接：arxiv.org/abs/2610.14625
为了更好地评估不同 Harness 与方法的自进化效果，我们做了 GDPevo Benchmark，涵盖 6 大真实企业场景，共 240 个任务。通过进行 Train-Val 划分，在防止过拟合的前提下公平评估 Agent 的自进化能力。
开源仓库：github.com/Prism-Shadow/GDPe…
论文链接：arxiv.org/abs/2608.03764
这条帖子不仅仅是介绍我们的工作，还希望能号召各位学界、工业界的伙伴来参与我们的开源项目。如果你也想在开源项目署名，共同发表论文，欢迎私信详聊～
A model that generates an image in four steps can still be painfully expensive to fine-tune.
Wasserstein-Tilted Flow Maps (yes, WTF) put that fast generation capability to work during training. A pretrained flow map can jump to a predicted final image, letting reward optimization skip expensive, multi-step rollouts.
The method rewards better outputs while penalizing how far the generation process moves from the original model.
In the paper's text-to-image experiment, WTF reached Flow-GRPO's peak reward with up to 280× less fine-tuning compute. The resulting model still supports few-step generation, without another distillation stage. Paper
That's the part I like: making a generator fast also gives us a cheaper way to improve it.
#NeurIPS2026 Spotlight
RL for diffusion and flow models often requires expensive rollouts.
We show that flow maps enable simulation-free RL finetuning.
Introducing Wasserstein-Tilted Flow Maps (WTF)! 🤩
The key is a transport-based alternative to KL regularization. 🧵
Saving an agent's conversation also saves the moment it decided the job was done.
The SLAs paper has a striking example. In one baseline run, agents repeatedly concluded their solutions were already optimal. Restarting resumed those same conversations. In the final 1,500 sessions, 98% made no tool calls.
Stateless Language Agents puts the experiment record in the harness. Every agent starts fresh, including the coordinator. The harness supplies relevant code and measured results, and the coordinator assigns specific experiments to workers.
On Anthropic's kernel task with Codex, SLA reached the strongest baseline's result with 93.1% fewer tokens.
I really like this approach. The system keeps what it has learned, while each new session gets a concrete piece of work to do.
AutoResearch agents keep burning tokens after they stop making progress. 💰😭
From authors of ACE and Meta-Harness: Introducing Stateless Language Agents (SLAs). Stateful search, stateless agents. Advancing the performance-cost Pareto frontier of RSI.
Across 9 tasks, up to 1B tokens:
- Best final result on every task 📈
- 14.5× fewer tokens to match the best kernel baseline 🚀
- Advisor (the “coordinator”) uses < 0.6% of total tokens 🫣
📑 Paper: arxiv.org/abs/2610.07625
NOT JUST HIGHER NUMBERS. Some more surprising findings in the thread below! 🧵 #LLM #AI #RSI #Agent
Yaowei Zheng | LlamaFactory retweeted
#NeurIPS2026 Spotlight
RL for diffusion and flow models often requires expensive rollouts.
We show that flow maps enable simulation-free RL finetuning.
Introducing Wasserstein-Tilted Flow Maps (WTF)! 🤩
The key is a transport-based alternative to KL regularization. 🧵
Really like this work. An agent shouldn't have to write everything out just to keep thinking.
Hybrid Reasoning Agents can reason in latent space, switch to text to call a tool, and then carry on with their internal context intact. That last part is what makes it interesting to me: it works across the back-and-forth of actually using tools.
The reported speedups are encouraging. I'd love to see this tested on longer, messier agent workflows.
Your LLM agent has to think out loud, token by token, before it can search, run code, or act in the world.
We humans don’t. We ponder in silence and speak when it matters.
Human language was never meant to be the limit of machine thought.
Introducing 🌗Hybrid Reasoning Agents
- Think in latent space, act through text
- Two modes, one unified multi-turn flow
- No black box, fully readable latent thoughts
- Any LLM can become a HybridAgent
Yaowei Zheng | LlamaFactory retweeted
How Do People Churn Out RSI Papers? An Unconventional Survey
We’ve published an unconventional survey of RSI that tackles a much-discussed question head-on: there seem to be quite a few “filler” papers in RSI these days. Where do people get their ideas for them, and how can we follow their lead to churn out papers of our own? We offer two approaches: combining different design dimensions, and borrowing from traditional deep learning.
First, we noticed that many RSI methods share similar underlying ideas, differing mainly in design choices such as what to change, how often to change it, and what feedback to use. We therefore developed a nine-dimensional taxonomy to systematically capture the commonalities and differences across method designs. All methods can be viewed as combinations of choices across these dimensions. From there, finding a new combination means finding a new idea.
The other approach is to rummage through the deep learning toolbox and see what can be brought over to RSI. Some RSI methods are already borrowing from deep learning: SkillAdam, for example, draws on the Adam optimizer to guide skill revision, while RRSI uses regularization to constrain the evolution of agent code. Familiar tools such as optimizers, regularization, and curriculum learning could all become the starting point for a new paper when transplanted into RSI.
We’ve compiled the relevant papers and their classifications on the Awesome RSI website, where you can browse and filter them by design dimension. Come find some inspiration, and feel free to submit a PR to the GitHub repository to add great RSI papers!
arXiv: arxiv.org/abs/2610.14625
Website: prism-shadow.github.io/aweso…
GitHub: github.com/Prism-Shadow/awes…
Zhihu: zhuanlan.zhihu.com/p/2093754…
Finally, we wish you all enough “filler” papers to graduate soon, so you can go do what you truly love.
Disclaimer
Expressions used in this post, including “filler,” “filler papers,” “churning out papers,” and “following their lead,” are informal, self-deprecating academic banter. They do not constitute a quality assessment of any specific paper, author, research group, institution, conference, journal, reviewer, research area, or any affiliated WeChat group. We are not saying that all RSI papers are filler, nor that RSI is a field of filler papers, nor that everyone working on RSI is just churning them out. In fact, the authors of this post are also working on RSI and have just published an RSI survey themselves, so any interpretation that somehow exempts us entirely from this criticism would be an overly charitable misreading. We acknowledge that RSI is an important, promising research direction that deserves serious study… (another 100 million tokens omitted).
Switching Opus 5.5 from a single agent to an agent team pushed the average cost per app from $24 to $122, roughly a 5× increase. Median completion time jumped from 75 to 174 minutes. In the same Vals AI test on 50 web apps, GPT-6 Sol's teams finished faster: 38 → 28 minutes. Both at max reasoning effort. Across four team-vs-solo comparisons, only one produced a statistically significant quality gain.
The leads organized the work differently. Sol delegated early and had the lead test the assembled app. Opus usually worked in sequential waves of builders and testers. Its max-effort teams spent two-thirds of their time after the first deployment.
Sol's biggest gains at medium effort came from building features and checking workflows that the single agent had skipped.
I'd use this to rethink when a harness delegates. A new agent should have work it can move forward independently, and the lead needs to check integration. Verification also needs a stopping condition; another round of tests can consume a lot of time without measurably improving the result.
vals.ai/blogs/multi-agent-vi…
This is probably the most useful little Mac app I've seen this year.
Tendedero hangs your screenshots on a clothesline at the top of the screen. Hover over the menu bar to reveal them, then click to copy or drag straight into another app.
You often need to send screenshots to someone or drop them into an AI chat. Keeping a few within reach, without digging through files on the desktop, makes a lot of sense—free, open source, and entirely local.
Adding a few "=" signs to a comment can make DeepSeek-V4-Flash-Base favor the wrong code completion.
In the figure, only the decorative comment length changes, yet the model's preferred completion switches between the correct "8" (green) and incorrect "32" (red) in a repeating four-token cycle.
The code is still there. What changes is how it gets packed into the model's memory.
To make long contexts cheaper, V4 compresses groups of tokens into fewer KV-cache entries. The paper finds uneven retrieval across positions in the compression cycle: some are much easier to recall than others. Shift the input, and you shift which positions hold the important details. In the code example, the effect repeats every four tokens.
For an agent, that suggests a frustrating failure mode. You can supply the right instructions and all the relevant context, yet a small change to the harness may change how reliably the model retrieves that context.
A single benchmark score won't tell you how stable this is. I'd want to see the same task tested across different token alignments, with the worst result reported alongside the average.
arxiv.org/abs/2609.36322
Jev helped open up a new direction for AI models. Less than a month after its launch, Microsoft, OpenAI, and Cloudflare are all shipping decision models or APIs.
An agent can use a strong model to make a plan, then keep calling faster models to decide what to do as conditions change. Cheap decisions let us put model calls in places where we used to settle for hardcoded rules.
My bet: these models will eventually handle far more calls than the big reasoning models. Executing one plan can require a lot of decisions.
Introducing Microsoft-Decision-1, our new model for fast decision-making.
It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality.
We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.
Jev's most interesting use case might be deleting code from agent harnesses.
We hardcode some decisions because asking a model at every branch is too slow or expensive. Make those calls cheap enough, and some of that if/else logic becomes optional.
I'd test the setup in the image: a strong model works on the plan, while a fast model checks whether the next action still makes sense.
In computer use, the page can change while the agent is thinking. A correct answer to the old screen can still produce the wrong click. Catching that change in time may matter more than thinking harder about the original plan.
TypeSafe AI's Diogo Almeida on why Jev is called Jev, and what cheap intelligence finally changes about software:
"It's wild that AI is so cool and software has been unchanged in 10 years... The most we can do is add a little chatbot in the side sometimes that can take actions, but not all actions."
"TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software."
"It's about the laziness to spend 10 hours to do the five-minute task instantly and to never have to do it again. It should be an ROI decision for people who automate stuff. I would just like it to be automatable."
"I think that people will just make new kinds of work, hence the Jev in Jevons. New kinds of work once that stuff is doable."
"As we have lots of intelligence at lots of different cost and speed trade-offs, the super systems-y types will be making trade-offs. Jev's gonna be 1000x too smart for them. They just want an approximate guess to optimistically route here and there."
Martin Casado: "The good news is we get to rebuild systems again. We have a new primitive. It's kind of a new way of thinking about doing software."
@CompleteSkeptic @typesafeai @bhorowitz @martin_casado
We may be overestimating how much a complex harness adds to a model’s capabilities.
If a workflow was added because an older model couldn’t handle something, we should reconsider it when a stronger model takes over.
If the model can already break down tasks, choose tools, and check its own work, how much of our scaffolding is still necessary? Could it keep adding calls and context without improving results?
What Does a Harness Buy? Tokens, mostly, got me thinking about this.
The authors held the model fixed and compared Claude Code, mini-SWE-agent, and OpenCode. Across 447 tasks and two Qwen models, Claude Code and the largely bash-based mini-SWE-agent were equivalent within a predefined margin of ±5 percentage points. The fuller framework showed no clear advantage in solving these tasks.
Costs differed much more clearly: with the same model and consistent pricing, the most expensive harness cost roughly three times as much per task as the cheapest. System prompts, tool definitions, and the number of calls all affected the bill.
Some harness complexity may persist because our assumptions haven’t caught up with the model.
An earlier model made mistakes, so we added a check. It struggled with a task, so we prescribed a workflow. Several model generations later, those designs remain, and every run keeps paying for them.
That’s a hypothesis, though. The paper doesn’t establish that stronger models necessarily need simpler harnesses.
But it suggests a useful experiment: with every model upgrade, try removing steps previously considered essential. See which tasks actually suffer. Keep recovery and context management where they help; let designs that no longer improve results disappear.
We’re used to describing a framework’s progress through what it adds.
Progress in the model should also show up in how much we can finally remove from the harness.
arxiv.org/abs/2610.04433
Odyssey has released Odyssey-3, a world model that generates interactive virtual environments and can help train control systems for robots and cars. It learns from internet video, gameplay, and simulated physical interactions to predict how scenes change in response to actions.
For example, Odyssey reports training a driving controller with 20 hours of driving data. The controller uses the pretrained world model to interpret what the car sees and predict where it should go. Only the controller is trained on those 20 hours; the underlying world model stays unchanged.
The goal is to reuse what the model has learned across different machines. Each new robot would still need demonstrations of its own controls, but could draw on physical knowledge acquired well before those demonstrations began.
odyssey.systems/meet-odyssey…
