The Open-Source LLM Evaluation Framework – created and maintained by @confident_ai GitHub: https://xitter.cf/t.co/kuvhlNRjo5
San Francisco
Joined February 2025
- Tweets210
- Following2
- Followers726
- Likes178
The best way to make use of tokens for production evals.
DeepEval retweeted
Today we’re launching Confident Monitor.
You probably have millions of traces from your AI agents, but they don’t tell you what’s actually broken.
Monitor reads your production traffic, turns failures into issues, and gives you a tested fix for each one.
DeepEval retweeted
Introducing Eval Skills: agent skills that help Claude Code and Codex build evals you can trust.
Start from real failures, establish a baseline, then let your coding agent improve your app one measured change at a time.
Open source, Apache 2.0. github.com/confident-ai/eval…
DeepEval retweeted
Evals started on ROUGE and BERTScores.
It evolved into LLM-as-a-judge because they weren't good enough.
Today, we're pleased to welcome back anti-language, LLM-as-a-judges.
Introducing Jev in @deepeval.
Full post in comments.
evals are important at all levels
I just made some new GBrain evals that help prove that my retrieval-for-AI-agent open source layer is SOTA for reading memory back without LLM-in-loop
And I've also added evals for memory-save from agent transcript, which is the best way to enrich your brain
Receipts at github.com/garrytan/gbrain-e…
New Breaking Change in v4.2.0: All metrics scores are now HIGHER THE BETTER.
No functionality changes, just something to be aware of for users on bias, toxicity, misuse, and knowledge retention metrics.
Changelogs: deepeval.com/changelog/chang…
Metric of the Week: ContextualPrecisionMetric
Your retriever can find the right chunk and still lose the answer.
This metric asks one question:
Did the useful chunks rank above the noise?
It scores the order of retrieval_context.
Each chunk is judged against the input and expected_output.
Relevant chunks near the top score higher. Bury them under noise and the score drops.
Use it to compare rerankers, top-k settings, hybrid search weights, and chunking changes.
Good for: labeled development and CI evals.
Not for: unlabeled production traffic.deepeval.com/docs/metrics-co…
Metric of the Week: TaskCompletionMetric
Agents can look busy and still fail.
This metric asks one simple question: Did the agent finish the job?
It looks at the trace and compares:
what the user asked for
what the agent actually did
If they do not match, the metric fails.
This should usually be your first agent metric.
Then add metrics that explain the failure: wrong tool, bad plan, or skipped step.
Good for: checking task completion.
Not for: judging only the final answer without tracing.
deepeval.com/docs/metrics-ta…
DeepEval is now the most used specialist agent eval tool at 17% in latest report from @VentureBeat
teams are finally taking evals seriously
yet almost half still shipped an agent that passed their tests and then broke with a customer
passing evals ≠ production ready
This is not just a client wrapper anymore.
The TypeScript SDK now includes:
> 47 of DeepEval's metrics
> 13 judge model integrations
> tracing
> component-level evals
> and integrations for @mastra, @vercel AI SDK, @OpenAI, @LangChain, LangGraph, and @OpenInferenceAI
Everything runs locally with a judge key like OPENAI_API_KEY.
No Confident AI account required. Add Confident AI later if you want shared reports and regression tracking.
Get started:
deepeval.com/blog/introducin…
