Skip to content

Sample the distribution. Then assert in tiers.

The main idea of pytest-agent-eval is first to run the function more than once, and what's the percentage of acceptable answers. The next question is "how do we measure an acceptable answer?"

# tests/test_booking.py

import pytest
from pytest_agent_eval import Expect, Turn

@pytest.mark.agent_eval(runs=3, threshold=0.66)
async def test_booking_confirmation(agent_eval, booking_agent):
    result = await agent_eval.run(
        agent=booking_agent,
        turns=[
            Turn(
                user="book me a slot tomorrow at 10am",
                expect=Expect(reply_contains_any=["confirmed", "booked"]),
            )
        ],
    )
    result.assert_threshold()
asciinema: pytest --agent-eval-live -vv, then again with -rP

Why -rP and not just -vv

pytest only prints a report section for a test that failed. A passing eval's per-run detail is recorded either way, but you have to ask for it: -rP is pytest's "report passed tests too". Without it, the run above is the single word PASSED — which is precisely the problem this page is about, one bit where you wanted a distribution.

TIER WHAT IT ASSERTS COST / RUN semantic a rubric, graded tone, completeness, reasoning an LLM judges the reply against words you wrote ~2 s · $0.01 variance: high structural what it did which tools ran, in what order, with what args ▸ the whole of the next page 0 ms · $0 variance: none deterministic what it said substrings and regex over the reply the same assert you already write 0 ms · $0 variance: none cost & variance Push every check down. A judge is a last resort, not a default.
svg: the three tiers, cheapest first

Look at distributions and distribute asserts in tiers

If a single sample cannot answer the question, take several and assert on the rate. runs=3 with threshold=0.66 says: run the whole transcript three times, pass if two of them do. The Run 2 ❌ in that output is the point, sitting under a test pytest reports as PASSED: a test that fails a third of the time is now a passing test, on purpose.

The second half is what you assert on each run, and there are only three kinds. Exact string checks on what it said. Structural checks on what it did. And a graded rubric for the things that genuinely need taste. They cost wildly different amounts and they compose freely on the same turn.

Ideally, a good test suite covers all tiers, where deterministic tests are more reproducible and cheap and can be run often.

Go deeper

  • Evaluators: every evaluator and how they compose
  • Python API: the agent_eval fixture, Turn, Expect, runs, threshold
  • Reporting: -v, -vv, and the markdown report