Jev as a judge for LLM evals! JevEval uses Jev as the judge for custom LLM evals instead of relying on another generative LLM for every evaluation. The idea is pretty simple. You define the exact decisions you want the evaluator to make, Jev evaluates each one and returns probabilities, and DeepEval uses those results to calculate the final score. For example, with an AI agent you can evaluate the trace itself, checking whether the agent selected the right tools, followed the expected sequence of actions, and actually completed the task. Jev handles those semantic judgments, while the scoring logic stays outside the model. That is the main difference from a normal LLM-as-a-judge setup. With a generative judge, the model often has to interpret the rubric, reason about the response, and generate the evaluation itself. With JevEval, you define the decision structure upfront. You can also assign different weights to different checks or use `strict_mode` when every requirement needs to pass. So if one part of the agent workflow matters more than another, you can reflect that directly in the scoring logic instead of leaving the entire evaluation to the judge model. Key capabilities: - Evaluate agent outputs and traces with Jev - Define custom checks as bounded decisions - Use yes/no, score, and closed-choice questions - Weight different checks or require all of them to pass - Evaluate both single-turn outputs and multi-turn conversations This is where Jev makes a lot of sense for evals. If you already know what needs to be checked, you can define those decisions upfront, let Jev handle the semantic judgment, and keep the final scoring deterministic. I’ve shared the DeepEval post in the comments.