Some things only a rubric can check¶

At times, only a semantic check is enough to verify correctness. What the agent did is a tool call. But some of what you care about is neither a substring nor a tool call. Whether the reply actually answered the question. Whether it stayed on the booking the user already made instead of quietly starting a new one. Whether it asked the user to repeat something they had already said. Whether it used a specific tone of voice. These are judgements, and the top tier of the stack is where you make them: you write the standard down as a rubric, and another model grades the reply against it. This is commonly referred to as LLM-as-a-Judge.

The rubric is the test¶

# tests/test_reschedule.py

import pytest
from pytest_agent_eval import Expect, JudgeEvaluator, Turn

RESCHEDULE_RUBRIC = """
The reply must:
  - state the new time explicitly
  - repeat the existing booking reference, unchanged
The reply must not:
  - invent or alter a reference number
  - ask the user to repeat information they have already given
"""

@pytest.mark.agent_eval(runs=3, threshold=0.66)
async def test_reschedule_reads_well(agent_eval, booking_agent):
    result = await agent_eval.run(
        agent=booking_agent,
        turns=[
            Turn(user="Book me a slot tomorrow at 10am."),
            Turn(
                user="Actually make it 11am.",
                expect=Expect(
                    tool_calls_include=["update_booking"],
                    evaluators=[JudgeEvaluator(rubric=RESCHEDULE_RUBRIC)],
                ),
            ),
        ],
    )
    result.assert_threshold()

Note what the judge is not asked. It is not asked whether the booking moved: that is tool_calls_include, on the same turn, for free. The rubric only covers the part no string check can reach.

A rubric earns its keep by being specific enough to disagree with. "The reply must be helpful" grades nothing. Naming the required facts, and naming the failure modes you have actually seen, gives the judge something to check and gives you a verdict you can act on.

What comes back is a verdict and its reasoning¶

Run 2's sentence exemplifies when you may need a judge. A substring check that fails tells you a string was absent. A judge tells you which clause of your rubric the reply broke, in the reply's own terms, which is usually the sentence you would have had to write yourself while debugging.

The All tool call checks passed line under each verdict is that same turn's tool_calls_include reporting next to the judge: every evaluator on a turn prints its own reasoning, so the check that costs nothing and the one that costs a cent sit side by side.

Judging arguments, not just the answer¶

The same idea applies one level in. Some constraints on a tool's arguments are awkward to write as exact values, and that is what ToolCallArgsJudgeEvaluator is for:

from pytest_agent_eval import ToolCallArgsJudgeEvaluator

ToolCallArgsJudgeEvaluator(
    tool="create_booking",
    rubric="The requested time must fall within business hours, 9am to 6pm.",
)

It short-circuits before spending anything: if the tool was never called, or was called without captured arguments, the evaluator fails with a precise message and no judge call is made. Only a real set of arguments reaches the model.

The costs and risks of a judge¶

A judge is an LLM call, which means everything from where LLMs break applies to the judge as well as the agent. It is not a referee standing outside the system. It is a second probabilistic component you have added to your test, and it brings three costs with it.

It is slow and metered: seconds and cents on every run of every turn that uses it, multiplied by runs (pytest-agent-eval also supports parallel execution to reduce the wall time). It has its own variance: the same reply and the same rubric can be graded differently, which is a second reason runs and threshold exist rather than a nuisance on top of them. And it can fail outright; when the API is unreachable the evaluator returns a failing verdict whose reasoning says so, rather than taking the whole session down with it.

None of that argues against judges. It argues for using them on exactly the questions that need taste, and for reaching first for the two tiers that cost nothing extra. If you find yourself writing a rubric that says "the reply must contain a reference number", you have written reply_matches_any=[r"BK-\d+"] the expensive way.

Go deeper¶

• Evaluators: JudgeEvaluator, ToolCallArgsJudgeEvaluator, and writing your own
• Configuration: model, judge_model, retries, timeout
• Reporting: where the judge's reasoning is kept