← all work
● case study 01evaluation2026

Benchmarking AI agents you can trust.

contextAn AI agent in a regulated domain
my roleEvaluation engineering, co-author of the benchmarking changelog
stackPython, LLM judges, versioned datasets
read moreEntry 1 ↗Entry 2 ↗
01 / the problemwhy

Choosing the model is a safety decision, not a preference.

The agent could run on any of two dozen frontier models, and those models often disagree on the same input. In a domain where every answer has to trace back to a source record, "it seemed better in testing" isn't good enough.

We needed a way to compare models on equal footing: the same agent, the same prompts and the same data, with only the model swapped. Scores had to be reproducible enough to stand up to scrutiny.

02 / scaleso far
0versioned, hash-stamped benchmark runs
0models evaluated behind one agent
0case-level evaluations
0trust dimensions every metric maps to
03 / approachhow

Score what users actually need, not how the model works.

Every metric falls under one of four questions: Is it correct? Is it grounded in the source? Could it cause harm? Will it hold up in production? That last one means commit rate, cost and latency are tracked on every run, not left for later.

  • Hold the agent fixed, swap the model. That turns cost, latency and accuracy into one comparable picture per run.
  • Score in a shared category space. The same condition can be coded a dozen ways, so both outputs and gold labels are mapped to a validated category index before scoring, not string-matched.
  • Report the strict and the lenient score side by side. One number either understates or overstates what the model got right, so we publish both.
  • Don't penalize the dataset's gaps. When a model surfaces a real finding the gold label never recorded, it gets credit instead of a false positive.
  • Count infrastructure failures as failures, not wrong answers. When a model's endpoint errors, it's logged, kept out of the quality score and retried.
04 / a resultbatch 1 → batch 2

A prompt rework, tested at 15× the sample size.

The first batch showed that recall, not precision, was holding every model back: models stayed quiet about real conditions far more often than they invented them. We reworked the extraction prompt and re-ran all 21 models on 300 cases. The first 20 cases were identical to the original batch, so the before-and-after comparison is clean.

Recall improved for all 21 models, by between +4.9 and +25.5 points. The cohort miss rate fell from 49.2% to 35.4%.

miss rate by category, pooled across 21 modelsbatch 1batch 2
Cohort overall
49.2%
35.4%
Heart failure
84.8%
32.4%
Coagulopathy
71.0%
59.0%
Liver disease
42.1%
32.4%

Lower is better. Batch 2 covers 300 cases × 21 models (6,300 case–model pairs). The 0.49% of pairs that returned an inference error are excluded and queued for retry.

05 / what we learnedtakeaways
0cost spread between cheapest and priciest model
0of recall that spread actually buys
0best recall, from a mid-priced model
0over-calling found on one rare category

The most expensive model was not the most accurate. One of the cheapest sat within a few points of the leader.

  • Small samples rank models badly. Rankings on a 20-case convenience sample reshuffled at 300 cases, so the next step is frozen, stratified validation splits rather than more of the same sampling.
  • Perfect sensitivity can hide a bad habit. On one rare, sensitive category, every model caught every true case but also asserted it 7× more often than it was present. That points to box-ticking rather than reasoning, and the fix is a narrow evidence-required rule.
  • Some failures aren't the model's. One model scored zero because its endpoint failed on every request. We flagged it and re-ran it instead of ranking it last.
  • Publish the uncertainty. Every entry says which results are solid and which are early signals.