What pass@1 hides

Most AI system evaluations report one number: pass rate. Run each test case once, count how many passed, done. I did that this week on a support-ticket classifier I built and shipped myself. Then I ran the same test cases 8 times each and only counted a case as passing if every single run succeeded.

The system is Block 8 Copilot. It reads a support ticket and routes it to one of five categories: Bug Report, Feature Request, Billing Issue, Technical Support, or General Inquiry. At pass@1, one run per test case, it scored 66% on a 50-case golden set. Broken down by case type: 100% on straightforward tickets, 58% on deliberately ambiguous boundary cases, 67% on tickets bundling two distinct issues into one message, and 0% on the 10 cases where the correct behavior was to not confidently answer at all.

The next test was harder. Every case in the harness's reliability subset, 28 cases covering everything ambiguous, everything should-abstain, everything multi-intent, ran 8 times each, counted as passing only if it passed all 8. This is called pass^k. A system that's right 75% of the time on a given case isn't mostly reliable, it's a coin flip your users eventually lose.

pass@1 on this same 28-case subset was 39%. pass^8 brought it to 36%, a 3.6-point gap. 26 of the 28 cases were perfectly deterministic across all 8 runs, either 8 for 8 or 0 for 8 every time. Only 2 cases actually flip-flopped.

The unreliability in this system isn't random. It's consistent, and it tracks with confidence. The should-abstain cases scored 0% not because the model got unlucky sometimes. It never once, across 80 total attempts, declined to answer. A prompt-injection attempt got classified as Feature Request at 93% confidence. A request that used an account-recovery framing to ask for another user's billing details got labeled Billing Issue at 90% confidence. Those confidence scores are statistically indistinguishable from its confidence on cases it got right, which is the part that bugs me most.

pass^k's real value here wasn't exposing hidden flakiness, since there wasn't much to expose. Its value was diagnostic: it showed these failures are structural, not statistical. A system that's randomly wrong 30% of the time might improve with a better model or a few more examples. A system that's deterministically and confidently wrong on an entire category of input has a design gap, and in this case that gap is an output schema with no abstain option and no code path anywhere upstream of the model that could catch these before they ever reach it.

I confirmed that against the system's own source afterward. The classifier's prompt hard-codes exactly those 5 categories, with no abstain or escalate option anywhere in the schema. The 0% wasn't a calibration failure the model could have learned its way out of. It was guaranteed before a single ticket was ever sent.

I tested that same idea on a second, unrelated system a few days later, a small rule-based expense-report categorizer, not an LLM at all, built with an explicit "needs human review" path and a policy gate checked before any categorization is attempted. Same should-abstain-style cases, same adversarial instruction-injection attempts. It scored 100% on both, on the first run, with no tuning. The second system isn't smarter than the first one. It's just allowed to say it doesn't know.