
evaluating Agents | prev GSOC @JuliaLanguage @mitacscanada GRI fellow
Local minima
Joined July 2017
- Tweets729
- Following1.6K
- Followers197
- Likes27.5K
I was trying to get a paper out, but it got rejected.
The problem statement is quite interesting Indic Scene Text VQA.
I think the rebuttal came down to having a small benchmark dataset.And i totally get it, but atleast you could appreciate the work for even starting the work in this direction, and have contribute some solid dataset and a decent benchmark set.
Oh well.
Paper: openreview.net/pdf?id=3Gyj19…
Deeptendu retweeted
man ignore the benchmarks and the race i just want there to always remain a frontier level intelligence model that is open and safe, available to everyone regardless of region
it is truly incredible to have built such an open (and strong) model with a group of genuine people
factorio is such a cool game/software. Training agents to write proofs for strategies is such goated nerd intersection of gaming, formalization and RL.
1
2
382
i have seen enough proof now that using a coding agent is a deep skill
it's confusing because the people you see heavily using them produce horrible results
but that's because it's a skill! you can get better and the ceiling seems pretty high - this is very exciting to me
Spoke a bit about building evaluation framework to testing our agents at @potpiedotai .As models become more and more capable, meaningfully evaluating them becomes becomes a hard challenge to solve.
Checkout our blog on how we setup our first evaluation datasets here: lnkd.in/gNpui5Nb
We are also starting to open source parts of our dataset. Checkout our @huggingface repo 🤗: lnkd.in/g9bai3Kk
The conversation of evaluating coding agents, usually begins with benchmarks like HumanEval, MBPP, or SWE-bench. And for good reason, because these benchmarks have played an important role in shaping how we understand model capability. They’ve given us a way to measure whether a system can write code, solve structured problems, and even work through real GitHub issues.
But over time, we started noticing where this approach begins to fall short. Most of these benchmarks operate in environments that are far cleaner than anything you encounter in practice. The tasks are well-defined, the repositories are easier to navigate, and the path to a solution is often clearer than it would be in a real system. It creates a setting where the agent knows what to do, which isn’t usually the case outside of benchmarks.
Enterprise codebases behave very differently. They evolve over years, shaped by multiple contributors, layered decisions, and dependencies that aren’t immediately visible. A small change in one place can spread across the system, and a bug might show up far from where it actually originates. In many cases, the right solution isn’t written anywhere... it has to be inferred from how the system has evolved.
That is the gap we wanted to evaluate, and it led us to shift the question. Instead of asking whether an agent can solve a coding task, we started asking whether it can understand a large, evolving codebase well enough to operate within it. That shift shaped how we built our dataset.
We worked with large open-source repositories that resemble enterprise systems, with deep histories and complex architectures. From there, we curated tasks that reflect real workflows. For QA, this meant problems where the answer isn’t in one place, and the agent has to connect different parts of the system. For code generation, it meant focusing on changes where writing code is only part of the problem, and fitting it into the existing structure is the harder part.
As models continue to improve, evaluation needs to evolve alongside them. Otherwise, we risk measuring progress on tasks that agents have already begun to outgrow, while missing the complexity that actually defines real-world use. For us, better datasets are not just about scoring agents, but about understanding what it takes to make them useful in real engineering environments.
In the video, @DSantra92 shares his perspective on why this evaluation gap matters and what it takes to measure agents in real engineering environments.
To see how we evaluate real-world engineering workflows, explore our evals and benchmark page:
potpie.ai/blog/the-agent-eva…
potpie.ai/evals
huggingface.co/datasets/potp…
2
162
wrote some rust 🎀
Every insight Potpie delivers about your codebase starts with parsing. Our context engine builds a complete map of your codebase: its structure, its components, its relationships into a knowledge graph that agents can use to navigate and query code faster. Currently, it supports more than 15+ languages including python, typescript, java etc. We understand that parsing large repositories is inherently time-consuming, but it shouldn't slow you down. That's exactly why we rebuilt this critical functionality in Rust: to make understanding your codebase faster, without any performance compromise.
Our benchmarks show approximately 30% faster parsing for repositories with 1M+ lines of code, with the performance gap expected to grow significantly for larger codebases. Looking ahead, we plan to introduce parallelization of the parsing pipeline to use multi-core threading for processing files simultaneously without Python's GIL constraints. This allows us to handle enterprise-scale repositories with sub-minute indexing times.
1
114
Deeptendu retweeted
We are announcing @potpiedotai $2.2M pre-seed fundraise to advance Spec-Driven Development for large enterprise codebases.
The round is led by @emergent_vc , with participation from @All_IN_CAPITAL , @DeVC_Global , and @PointoneCapital , along with the support of some amazing angel investors from companies including Atlassian, OpenAI, Meta, Razorpay and Flobiz.
As AI accelerates code generation, the constraint inside large enterprises has shifted from coding to maintenance and assurance. The limiting factor is no longer writing code, but understanding complex systems, aligning teams around intent, and safely evolving large, interdependent codebases.
In most organizations, specifications exist as static documents, while production systems evolve independently. Context is fragmented across repositories, tickets, logs, reviews, and floating documents making reliable AI adoption difficult.
We are building the foundational layer that makes Spec-Driven Development executable at scale.
By unifying engineering context and operationalizing the spec as a structured source of truth, we enable AI systems to reason with architectural awareness rather than surface-level code completion.
We are already working with large enterprise customers, including Fortune 500 organizations. This milestone allows us to deepen those partnerships and support more teams transitioning from experimental AI usage to structured, production-grade AI-first engineering workflows.
If you are leading engineering at scale and evaluating how AI should integrate into mission-critical systems, we would love to chat with you!
codex is by far a better coding model than opus - anyone who knows anything understands this
but the whole industry should reflect on why opus is the most popular
people assume whatever is the smartest will win but the old rules of product are still what determine everything
Introducing GLM-5: From Vibe Coding to Agentic Engineering
GLM-5 is built for complex systems engineering and long-horizon agentic tasks. Compared to GLM-4.5, it scales from 355B params (32B active) to 744B (40B active), with pre-training data growing from 23T to 28.5T tokens.
Try it now: chat.z.ai
Weights: huggingface.co/zai-org/GLM-5
Tech Blog: z.ai/blog/glm-5
OpenRouter (Previously Pony Alpha): openrouter.ai/z-ai/glm-5
Rolling out from Coding Plan Max users: z.ai/subscribe
1
107
Anthropic testing out new models by building new products?
Claude in Excel is now available on Pro plans.
Claude now accepts multiple files via drag and drop, avoids overwriting your existing cells, and handles longer sessions with auto compaction.
Get started: claude.com/claude-in-excel
2
119
lessggooooo
GLM-4.7 is here!
GLM-4.7 surpasses GLM-4.6 with substantial improvements in coding, complex reasoning, and tool usage, setting new open-source SOTA standards. It also boosts performance in chat, creative writing, and role-play scenarios.
Default Model for Coding Plan: z.ai/subscribe
Try it now: chat.z.ai
Weights: huggingface.co/zai-org/GLM-4…
Tech Blog: z.ai/blog/glm-4.7
1
163
If someone makes this for Indian Railways, they will get rewarded with court cases from the government.
A cracked Japanese student has built an absurdly detailed JR (and China Railway) ticket generator. The amount of buttons and knobs on this thing is insane. It's like a train+software+weeb fidget board.
1
239
























