Evaluation Harness. The Foundation

Why this block exists

Scheduled here because it has zero upstream dependencies. Everything measured before a measurement layer exists is retrofitted and unverifiable, and that's the single biggest correction I made to the plan's original ordering. The bet: build the ruler before the thing it measures, so the golden set can't drift toward whatever the system under test already happens to handle well.

What I built

agenteval, an evaluation harness for AI agents, as a real Python package instead of a one-off script: HTTP and CLI adapters, a golden-set format that requires labeling rationale (not optional), Wilson confidence intervals on every reported rate, pass^k reliability testing, and a cross-model LLM-as-judge pipeline that has to validate against independent human labels before I trust it for anything.

What I did

Week 1 was methodology before touching real data: the golden-set spec, a prediction sheet, a first interface-contract sketch. The proof that the delay was worth it came from an old 22-case classifier sitting around from an earlier block. Running a Wilson interval on its "95.5% accurate" result turned that into "somewhere between 78% and 99%." Not the same claim at all.

Week 2 built the harness for real and pointed it at the actual target: Block 8 Copilot, a production support-ticket classifier. 50-case golden set across 5 case types, coverage runs, reliability runs, first real numbers back.

Week 3 added the piece the schedule had pushed off: an LLM judge (Claude Sonnet, deliberately cross-model against Copilot's GPT-4o) validated against independent human labels using Cohen's kappa. I also built a second worked example, expense_router, specifically to check whether the fix Copilot needs actually holds up when someone builds it for real.

What came out of it

Copilot scored 66.0% pass@1 (95% CI: 52.2%–77.6%). The number underneath that mattered more: 0% on all 10 should-abstain cases, fully deterministic across all 80 pass^8 attempts. I checked the system's own prompt directly. It hard-codes 5 categories, no abstain option anywhere in the schema. It's not that the model failed to learn restraint. It was never given anywhere to put an answer it shouldn't guess at. A prompt-injection attempt landed as "Feature Request" at 93% confidence, statistically no different from its confidence on cases it got right.

The pass@1-to-pass^8 gap was smaller than I expected too: 39.3% to 35.7%, a 3.6-point gap on the 28-case reliability subset, with 26 of 28 cases perfectly consistent across all 8 runs. So Copilot isn't randomly unreliable on hard cases. It's consistently and confidently wrong on an identifiable subset. That's a design gap, not a tuning problem.

The judge validated at 84.0% raw agreement with independent human labels, Cohen's kappa = 0.654 ("substantial"), and every disagreement clustered in the case types where you'd actually expect judgment-call variance (ambiguous, multi-intent), not the unambiguous ones. Getting to that number meant catching my own mistake first. The first human-labeling pass scored all 10 should-abstain cases "pass" even though the written reasoning argued the opposite. Uncorrected, that would've produced a kappa of 0.060 and made the judge look nearly useless. I caught it by checking whether the stated reasoning actually matched the score before trusting the labels. No automated check flagged it.

expense_router, a second and deliberately different system (CLI instead of HTTP, deterministic rule-based instead of an LLM), tested the fix Copilot needed: an explicit "needs human review" path checked before any categorization is attempted. 100% on should-abstain and adversarial cases, first run, no tuning. Same harness, zero code changes to run it against a completely different kind of agent.

What I'd change

Build Copilot's rejection path as deterministic pre-processing ahead of the classifier. Pattern-match empty/gibberish input, injection markers, and PII-disclosure requests before the ticket ever reaches the model, and route those straight to human escalation regardless of what the model would've said. Should-abstain scored a fully deterministic 0%, so this isn't a better-prompting problem. It needs an explicit code path, which is exactly what expense_router proves out.

What carries forward

The evaluation discipline built here (labeled golden sets with mandatory rationale, Wilson intervals on every rate, pass^k for anything claiming reliability, cross-model judges validated against humans before they're trusted) is now what every later block gets measured against. The should-abstain finding also changes what Block 11's advisory bot has to prove before it goes near a real repository. Giving good suggestions isn't enough if it has no way to say it doesn't know.

Full write-up, including the actual numbers, disagreement detail, and the LinkedIn version: see the artifacts above, or "What pass@1 hides" in Writing.

Decided not to automate. A second independent labeler on the ambiguous and multi-intent cases, the ones where the judge and my single human labeler actually disagreed. One labeler is enough to catch a labeling error (which happened, see surprised below) but not enough to know whether the golden set's labeling_rationale field actually holds up when someone who didn't write it has to defend it.

What surprised me. How small the pass@1-to-pass^8 reliability gap actually was: 3.6 points, 26 of 28 cases perfectly deterministic across all 8 runs. I expected repetition to expose hidden flakiness. Instead it mostly confirmed the opposite: Copilot's failures are structural and repeatable, not random, which changes the fix from "improve the model" to "add a code path that doesn't exist yet."