Skip to content

Where LLMs break

prompt agent pass / fail distribution of replies assert one sample PASS FAIL …both, at random 1 · deterministic temperature, tools, model updates 2 · single-valued "you're all set" is also correct 3 · exact substring checks are a proxy, not a spec 4 · cheap ≈ 4 s · $0.02 × every run You are not asserting on an output. You are asserting on a property of a distribution, from one sample.
svg: the previous page's geometry, broken.

Now imagine you have a function that calls an LLM:

# tests/test_naive.py
from pytest_agent_eval.models import Message

async def test_naive_booking(agent):
    reply, _ = await agent([Message(role="user", content="book me a slot tomorrow at 10am")])
    assert "confirmed" in reply
asciinema: the same assert, five processes, failing twice

LLMs are probabilistic

Run the naive test five times and it fails twice - same code, same prompt, same commit - not because the agent misbehaved, but because "you're all set" and "reserved" are perfectly good confirmations that happen not to contain the word we picked. Every one of the four properties is gone at once, and they took the binary outcome down with them.

LLMs are designed to predict the next likely token. However, especially for closed models, we cannot guarantee reproducibility for the models (even setting hyperparameters). In the end, you are looking at a sample, not the distribution. A single run cannot answer a question about a distribution's behaviour, and no amount of staring at that one run will fix it.

Running it many more times costs seconds and cents per call, and we would still miss how many results are acceptable or not. What can we do?

Go deeper