NLP researcher @MegagonLabs
United States
Joined January 2020
- Tweets70
- Following494
- Followers178
- Likes176
Pinned Tweet
๐ข New preprint from @MegagonLabs ๐งต
Confidence estimation for LLM agents is hard: evidence of success spans tool calls, observations, reasoning, and memory. When model internals are inaccessible, reruns are expensive, and training data is scarce, how can we reliably assess whether an agent succeeded and how much to trust its output?
Meet ๐๐ผ๐ป๐ณ๐ถ๐ฑ๐ฒ๐ป๐ฐ๐ฒ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐๐ฟ๐ฎ๐ฝ๐ต๐ (๐๐ฅ๐): a training-free, black-box method that turns a single execution trajectory into a structured, evidence-grounded argument for success and an auditable confidence estimate.
Paper: arxiv.org/abs/2610.07948
Code: github.com/megagonlabs/crg_cโฆ
Can you audit a confidence score?
With CRG, yes:
(a) the graph shows how many likely-true conditions add up to a cautious estimate
(b) CRG misses that the agent fixed a rejected tool call, and a user can trace the low score to that one claim and override it
CRGs make confidence estimation for LLM agents practical without white-box signals, repeated runs, or labeled data, and each estimate comes with the evidence behind it.
Paper: arxiv.org/abs/2610.07948
Code: github.com/megagonlabs/crg_cโฆ
Work led by our amazing intern Brendan King. Kudos to my co-authors Jean-Flavien Bussotti, @PPezeshkpour, and @estevamhruschka at @MegagonLabs!
๐ข New Preprint
Prompting is a flexible way to adapt LLMs, but prompt engineering is still manual and hard to scale. Can function-calling LLMs act like prompt engineers: inspect failures, diagnose patterns, and revise prompts?
Meet Reflective Prompt Tuning (RPT): a diagnosis-driven framework for iterative prompt tuning with function-calling LLMs.
Paper: arxiv.org/abs/2605.21781
@MegagonLabs
1/n
Do prompt patches predict improvements?
For reasoning-heavy tasks, patches for step-by-step reasoning, output validation, and confidence calibration are associated with better performance and calibration.
For domain-specific numerical reasoning, gains are more mixed, suggesting some errors may require more than prompt edits alone. 7/n
See our paper for more results and findings. Kudos to my co-authors @moinnas @PPezeshkpour @estevamhruschka
Paper: arxiv.org/abs/2605.21781
Code: github.com/megagonlabs/RPT