Skip to content

Gates and statistics

[tool.agent_eval.groups.booking] tags = ["gate:booking"] edge_case ▲ must_pass booking_confirmation 9 / 10 = 90% ≥ 90% required PASSED exit code 0 failure absorbed by the gate, still printed in the summary Three runs is not a hypothesis test. It is a tripwire, and gates are where the power comes back.
svg: a group threshold with a must_pass pin
# pyproject.toml

[tool.agent_eval.groups.booking]
threshold = 0.9                          # 90% of matched tests must pass
tags = ["gate:booking"]                  # match transcripts by tag
must_pass = ["booking_confirmation"]     # this one must individually pass

[tool.agent_eval.groups.smoke]
tags = ["smoke"]                         # threshold defaults to 1.0
asciinema: a group gate absorbing one failure, and exit code 0

Predictability vs. flexibility

A suite of probabilistic tests may fail somewhere almost every run. We may test for specific words, but we cannot cover all possible combinations (or would be extremely expensive). The idea is that the 90% that passes gives us enough indication that the remaining 10% is "ok". If any red turns CI red, you will be dealing with many false positives. So gate on the aggregate: 90% of booking evals must pass, with specific ones must pass individually. Everything else may flicker, and every failure is still printed — the run above ends 1 failed, 13 passed below the override line, because the override changes the process exit code and not the report.

One may argue that: three runs at 66% is not statistically significant. Correct. It is not a hypothesis test and should not be read as one: you are not estimating a true pass rate, you are placing a tripwire that catches large regressions cheaply.

However, a gate over forty evals aggregates hundreds of runs, and that number moves meaningfully when something breaks. Per-test thresholds suppress noise; the group gate is the number you actually trust.

Go deeper