Surface Zero Goes Live

Why this block exists

Scheduled here because every posting in this category screens on operating experience over time, and there's no later date at which that clock can be caught up. A thin system with real history beats a sophisticated one with three weeks, and Block 11 is the only place in this plan that clock could even start.

What I built

Surface Zero: a comment-only advisory agent that reads a new issue, suggests a label from the target repo's own set, flags a likely duplicate, and does nothing else. No write access beyond that single comment. The build follows Block 10's discipline into a live setting rather than replacing it: a ledger writes a started row before the agent runs and ends in exactly one immutable terminal state, so any run's actual outcome is checkable after the fact instead of trusted on faith. A kill switch gates the outbound comment specifically, never the model call, so disabling posting still leaves a record of what the agent would have said. And abstain is a first-class action with its own tool schema entry, not a fallback, carrying Block 10's should-abstain finding forward: a system with no schema slot for "I don't know" will eventually say something confidently wrong instead.

What I did

Outreach came before any of that got deployed. PCNtoolkit was the first real ask, chosen for a genuine neglected-backlog signal, and it declined on 2026-09-14. A second round (Amethyst-Mod-Manager, MetaHookSv) and a third (orchestkit, a stronger thematic fit) followed the same real-ask discipline; orchestkit went ten days without an answer. Rather than let a stranger's yes that might never come slip the block's own deadline, the target became ForgeAS/everForge, a repo I own myself, named plainly as a substitution rather than treated as equivalent to consent.

The pipeline came together target-agnostic first. Fly deployment went clean on the first attempt, no config correction needed. The kill switch got tested live in both directions against the running instance, not just against unit tests, with one real methodological catch along the way: a test payload that described itself as a kill-switch test got correctly abstained on as too ambiguous, so the first attempt never reached the code path it was supposed to exercise. Getting a real webhook delivery working end to end surfaced the block's actual finding: the posted comment read as me replying to my own issue, not as a distinct bot. A fine-grained PAT, correctly scoped to Issues-only exactly as designed, still authenticates as whoever generated it, and a permissions-only check has no way to catch that. Fixed by migrating to a GitHub App with its own installation-token identity, re-verified live as surface-zero-advisor[bot].

With the identity fixed, the blind test kit ran: 16 issue scenarios generated by Gemini from a domain brief that never mentions labels, categories, or triage, so the generator itself was blind to the fact that classification was being tested at all, not just working from an answer key written in advance. All 15 non-abstained results posted for real, through the same pipeline the live path uses.

What came out of it

15 of 16 exact matches against the answer key, one defensible abstain rather than a wrong guess, both engineered duplicate pairs caught with the correct originals pointed to. The single case worth naming: issue #8 was deliberately written to bait a bug label onto a request for functionality the target app never claimed to have. The advisor reasoned from what the app actually does rather than the reporter's framing, and called it enhancement instead of taking the bait, which is exactly the failure mode that kind of test exists to catch.

The bigger finding wasn't a number at all. The scope check on the posting credential passed every test it had and still missed the identity problem, because scope and identity are different properties of a credential and nothing in the methodology up to that point had asked a test to check the second one. That's a different category of gap than a failing test: some things don't fail a test, they fail to be tested in the first place, and this one only surfaced because a live comment got read, not because anything audited it.

What I'd change

The outreach thread should have had a check-in date attached to each round from the start, rather than each round's silence getting discovered mainly in retrospect when the next deadline forced the question. Orchestkit's ten days of silence weren't flagged until well after they'd passed. And the prediction sheet's own discipline slipped a third time in this block specifically: slot 5 was left as a placeholder and never filled before the live test happened, the same shape of miss Block 10's retrospective already called out once. A pattern that recurs after being named explicitly isn't a one-off anymore.

What carries forward

Live, continuously-updated version of these numbers -- population by evidence class, cost per successfully evaluated recommendation, the consequence-gate statement, and the identity-attribution incident -- lives on the Surface Zero operating page, not just in this writeup.

The scope-versus-identity distinction is now its own section in the methodology doc, not a fixed one-off bug: the next credential this program hands to anything posting publicly gets both questions asked, not just the scope one. The blind-test-kit pattern, a generator blind to the test rather than just an answer key written in advance, is reusable past this block for any agent whose output needs checking against unseen, realistic input instead of fixtures the builder already knows the shape of.

What this can't say yet: whether batch 2's harder, by-design-ambiguous cases hold up the way batch 1 did, what a Wilson-style interval on live accuracy would actually show against an unbounded issue stream instead of a fixed offline set, and what Gate A's real operating history looks like past a batch-posted test run. Also worth being plain about: the ladder's rung A still correctly shows no reached date. Giving good suggestions against my own repo isn't the same claim as a stranger consenting to run this against theirs, and that's the actual gate this block hasn't cleared yet.

Decided not to automate. Nothing in the automated test suite caught the credential identity gap, because the scope check tested exactly what it claimed to test, correctly. Scope and identity are different properties of a credential, and the methodology hadn't asked the tests to check the second one. Found by reading a live comment's byline, not by any check that ran on its own.

What surprised me. How completely a correct, fully-tested scope check still missed the identity problem. I expected a security gap, if there was one, to show up as an authorization failure the tests could catch. Instead the token did exactly what it was scoped to do and still posted under the wrong identity, which is a different category of miss than a failing test: some gaps don't fail a test, they fail to be tested at all.