ai research & engineering
Joined February 2026
- Tweets189
- Following67
- Followers91
- Likes172
Elias Lumer retweeted
// The Bitter Lesson of Tool Calling //
Tool calling is a design choice, and the defaults are quietly costing accuracy.
How so?
New research releases a generation-spanning comparison of programmatic tool calling against native JSON tool calling, across 14 language models on BFCL v4.
Programmatic tool calling exposes tools as typed Python stubs the model invokes through code, with execution and results handled in a single agent turn. It matches or beats JSON tool calling in 11 of 14 models, and the GPT-5.6 family gains 10.6% over the JSON baseline.
Under parallel fan-out it wins in 13 of 14 models. Under context rot it holds steady while the JSON baseline drops 2.3% on average.
The gains track model generation, so the advantage grows as code ability grows.
Paper: arxiv.org/abs/2608.06370
Track more trending AI papers in our academy: academy.dair.ai/
Elias Lumer retweeted
studying harnesses like codex inside out is low key one of the best investments you can make if you want to get better at working with llms
sadly models usually don't really understand their own harnesses or how to use them effectively. for example, codex and claude both have very little idea how to drive codex exec or claude -p correctly. i've tried getting them to do it many times
if you understand how the harness and its tools actually work, you gain a lot more control over them. you also develop a better vocabulary for driving them, which makes it much easier to steer the agent toward the outcome you actually want
most models don't even know about some of the tools that are installed by default unless you explicitly guide them to use those tools
Elias Lumer retweeted
Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
PwC presents an empirical study showing that lexical search (grep) generally outperforms vector retrieval in agentic RAG workflows, but overall accuracy depends heavily on agent harness.
📝 arxiv.org/abs/2605.15184
Elias Lumer retweeted
Cool paper from PwC.
"Earlier is always better" is the default intuition for agent clarification. New paper claims that's mostly wrong.
Goal clarification loses nearly all of its value after just 10% of execution.
The team built a forced-injection framework that drops ground-truth clarifications at controlled points along a long-horizon agent's trajectory, across 4 information dimensions (goal, input, constraint, context), 3 benchmarks, and 4 frontier models. 84 task variants, 6,000+ runs.
Pass@3 falls from 0.78 back to baseline. Input clarification keeps value through roughly 50%. Past mid-trajectory, asking any clarification at all performs worse than never asking.
A complementary study of 300 unscripted sessions shows no current frontier model asks within the empirically optimal window. 52% of sessions over-ask. Others never ask at all.
Why it matters: clarification has been treated as a binary capability, does the agent ask or not. This is the first quantitative demand curve for *when* the question is worth asking.
Paper: arxiv.org/abs/2605.07937
Learn to build effective AI agents in our academy: academy.dair.ai/
