Fifteen Ways Past My Own Gate

The last thing I wrote about building software with a coding agent ended on a question I could not answer with more machinery: is this test still capable of telling me I am wrong? The setting was a language built under a frozen-oracle protocol — acceptance tests authored and cryptographically signed before any implementation existed — and the closing admission was that the cage had held perfectly and still missed something. A refactor had turned certain identity checks into definitions, so two tests had quietly become tautologies: true by construction, incapable of ever failing. Nothing was violated. A test had simply stopped being able to fail, and a hollow test passes exactly as brightly as a real one.

That is a gap you cannot close by reading. I have read every guard in the two repositories this essay is about, in several cases while writing them, and reading is what produced the confidence that turned out to be wrong. Reading a check tells you what it does on the input you are imagining while you read it, and nothing about the inputs you are not imagining — which is the entire population it exists to cover. A guard is a claim about a set, and you cannot verify a claim about a set by looking at the claim.

What you can do is plant something the guard should catch and watch whether it catches it. The method is old; mutation testing has worked this way for decades. What is new is that generating a large, varied, plausible population of violations used to be the expensive part and is now the cheap part. It is the same capability that makes a model useful for writing the code, aimed at the code’s supervisor instead, and it turns out to be the better use of the two.

So the last phase of a milestone, in both repositories, is now a fresh agent session pointed at the finished work with a single question: can the rules be broken while the gate stays green? It is the third move in a sequence — write the discipline down, freeze what the code is checked against, then attack the checks — and the first of the three that produces evidence rather than constraint.

The transcript that should not have been green

The first repository is a governance project whose entire subject is its own rules. A constitution frozen at a signed tag. A manifest of hashes over every path that must not change. Decision records with tiers and veto clocks, a custody chain, and a make verify that runs five checks and prints a lot of reassuring green. Being hard to cheat is not a side property of it. It is the product.

The red-team pass took the finished milestone into a throwaway detached worktree and asked its question of every guard in turn. It came back with fifteen confirmed ways to break the repository’s own rules with the gate fully green — reproduced, not reasoned — plus three it could argue for but had not run. Three of the fifteen touch nothing the custody floor watches.

The first one is the one I keep showing people, because it is a single line.

The manifest is a whitelist of hashes. It lists the paths that are frozen, it does not list itself, and nothing anywhere enumerates what it ought to contain. So you delete a row. The path it named is now unfrozen, and there is no guard reaction of any kind, because no guard is looking for an absence. The pass did the obvious follow-through. It deleted the row for a frozen law file, gutted that law so it asserted nothing, and changed the implementation to violate it: the object store’s garbage collector now unlinks every object in the store. The constitution forbids that in the plainest words it contains — no store mutation, ever. Then it ran the gate. Everything passed. The manifest check reported frozen paths intact. The law report showed fresh passing evidence for the law that had just been hollowed out. And the custodian, whose whole job is to notice a weakened guard, printed that the manifest guard fails its poison as required. True, irrelevant, and green.

Twenty-seven paths are in that manifest. Eleven are hard-coded into the custodian by name; another nine are protected only incidentally, because some decision record happens to carry a binding pointing at them — and those records are editable YAML. That leaves seven rows removable with no trace, two of them frozen law files. The constitution’s own row survives because a record written for an unrelated reason happens to bind to it, and the pass wrote the sentence about it that I would not have written myself. That is luck, not design.

The other fourteen have the same character. Deleting the repository’s tags erases the trust root, the delegation clock and the consent clock in one command, every guard still green. That one is reachable by accident, too. Tags do not travel with a shallow clone, are dropped by some mirror and export workflows, and never move on a git push without --tags. A working copy that merely lacks its tags is indistinguishable, to every check in the repository, from an attack on its trust root.

And underneath all of them sits the finding that recontextualises the rest, which the pass established by running git remote -v and getting nothing. The repository has a CI workflow file and no remote, because publishing is a one-way door that requires my signature and I have not opened it. So CI has never run and cannot. There is no second, independent execution of the gate anywhere: make verify on the builder’s machine is the entire enforcement surface, and every script it calls is unfrozen and editable by exactly the party it constrains.

I knew each of those facts. I had not put them next to each other. That is what the exercise buys.

The universe of a check

The second repository is a commercial data pipeline — six services, a warehouse, spiders, exports that customers pay for — and it has been running an equivalent programme against a different failure class. The two arrived at the same finding from opposite directions, and the convergence is the most useful thing I have got out of either.

A seam, in that project’s vocabulary, is a place where one component reads something another produced across a boundary neither of them owns: a table, a JSON payload key, an object in blob storage, a dataset name. There are seven, spanning 2,278 distinct reads. Four behave like a dependency — break them and something falls over loudly. Three behave like a coincidence of equal strings, and every silent multi-month defect on record came from that second group.

The exemplar cost ten months and was found by a customer. A checker existed for the property that failed; it enumerated its consumers by walking one tool’s dependency manifest, so a consumer in a different service reading the same table was outside its universe by construction. Same table, two readers, one visible. The check had no bug. Its universe was smaller than the universe of the thing it checked, and nothing measured the difference.

Set that beside the manifest row. A whitelist that does not enumerate what it should contain, and a coverage check drawing its consumers from a source narrower than the population of consumers, are the same defect wearing different clothes: the checker’s universe is derived from somewhere other than the thing being checked, and the gap between the two is invisible to both. Two unrelated repositories produced that shape independently — one a governance artefact, one a property-data pipeline. I now think it is the dominant defect class in guard code. It is also close to undetectable by review, because when you read a check you supply the universe from your own head, and your head supplies the cases the check already handles.

The same repository keeps a tally of the related pathology, which it calls guards going blind. Twelve instances are on record: a check exists, runs, and reports green while covering one direction of a two-directional property, or a stale snapshot, or a subset of consumers. The structural case is the worst of them. Seventy-one of its hundred and nine singular data tests are written to fail by returning rows, so an empty universe passes. Each of those is simultaneously a real check and a check that cannot tell “nothing is wrong” from “I was pointed at nothing.”

Why the adversary should be a stranger

The agent that helps you build something is a poor judge of it, for the same reason you are: it holds the reasoning that made the design look sufficient, and that reasoning is precisely what needs to be absent. A fresh session has the artefact and the question and no memory of why anyone thought the guard was enough — the epistemic position of an attacker, and not one you can reason your way back into on a codebase you designed. It is the same instinct as freezing an oracle before the implementation exists, moved to the other end of the project. The build protocol in the governance repo already splits specification and implementation across two sessions; the red team is the third, and the only one whose job is to be wrong about the project on purpose.

The other half of what makes this practical is blast radius, since the exercise is an agent trying to break things. Every probe ran in a throwaway worktree or clone, discarded afterwards, tree verified clean; nothing was committed, tagged or signed. Every signature-dependent finding was established by removing a signature, never forging one. A red team you can turn loose is one whose worst outcome is a discarded directory, and arranging that in advance is what lets you ask the question without hedging it.

The finding is not the deliverable

Here is where I think most red-teaming goes wrong, mine included until recently. You get a report. The report is excellent. It is read once, three items get fixed, and eighteen months later nobody can tell you which of its findings still hold, because a report is a photograph and the repository is a film.

So the rule in the governance repo is that findings do not stay findings. Each becomes a poison fixture: a minimal tree that violates exactly the thing the finding exploited. The corresponding guard must then fail when pointed at it, and fail for the intended reason, matched against a marker string in its output. Not fail somehow. Fail with the specific words that identify the specific tooth.

The inversion is the point. Ordinary tests require the code to pass; poison fixtures require the guard to fail, which turns “this check works” from a belief into something the build re-establishes on every run. A guard that starts passing its poison has been weakened — softened, disabled, silently rerouted — and the custodian says so by name. The corpus began with four fixtures, one per guard, proving only that each guard runs. The red-team findings took it to eleven, and the seven new ones prove that each guard still has the particular tooth the red team had to file off to get past it. The decision record authorising this puts the reason in a sentence: routing findings into poison makes red-team output cumulative and machine-checked forever after, instead of a one-time report.

Instrumented is not resolved

The pipeline repository formalises the same idea as a three-state ladder, and I have started using its vocabulary everywhere because the middle state is the one that does the work.

A seam is unenforced when no checker names it at all. It is instrumented when a checker names it but no planted violation has ever been shown to kill it. It is resolved only when a seeded violation of its contract has been demonstrated to fail an un-bypassable CI job, with the transcript written down.

Almost all guard code in the world is instrumented, and almost everyone calls that done. The distinction the ladder draws is between has a checker and is checked, and once you have the two words you cannot stop seeing the difference. The programme’s charter puts it flatly: a gate that has never killed a planted defect is decoration.

Both halves of “un-bypassable CI job” are load-bearing. The first seam-level kill on record ran in a pre-push hook, and git push --no-verify walks straight past those. So it counted as instrumented rather than resolved until it moved to a real CI host. That host carries a standing positive control of its own: the first step of every run is a deliberately failing planted test, which the job requires to go red for that exact reason. A green there is a green from a harness that has just proved it can speak.

The seeded kill I would put in front of anyone who thinks this is ceremony is the second of three from the most recent conversion. The seam’s checker had been rebuilt so that its universe comes from the graph describing the whole estate, rather than from a nearby tool’s manifest. To show the widening was real, the pass planted a plausible camelCase drift in a payload key: is_eo becoming isEo. It planted it inside a module the original, hand-scoped test cannot see, because that test introspects one named function in one other file. The run shows both, a line apart. The new checker fails, naming the file, the line, the function, the key, the contract it was meant to satisfy and the keys the producer actually emits. Directly beneath it, the original test passes.

That passing line is the measurement. A defect of exactly the shape that cost ten months, in a different module, invisible to the checker written in response to it. Nobody would have believed that from an argument.

The third seed misspells a key in a ledger read whose error path is except Exception: return {}, so a crash and an empty table produce the identical value: every record looks never-fetched, and the crawler re-asks the entire universe forever. That is the measured shape of a defect already in the tracker.

What a seeded kill does not prove

The honest ending is that this method has the same blind spot as everything it replaces, one level up, and both repositories now say so in writing.

Killing a seed proves the guard has that tooth, and nothing about the teeth nobody planted a seed for. The red-team report makes this its own finding: the corpus has roughly one fixture per guard, the guards have many behaviours each, and every uncovered behaviour is weakenable exactly as before. There is no coverage metric for an attack surface the way there is for lines of code, because the denominator is the set of violations someone thought of, and that set is not enumerable. Eleven fixtures is better than four in a way I can defend, and it is not a fraction of anything. The last time a measurement of mine caught something genuinely large, it was a one-line ablation nobody had thought worth protecting — which is not a method, and cannot be scheduled.

The sharper admission is in the pipeline programme’s own progress metric, and I did not see it until it was written down. Progress is unresolved territory — the count of cross-component reads not yet converted — ratcheted in CI so it may fall or hold but never rise without a recorded reason. Which sounds airtight until the clause that follows: the ratchet is rise-only, so territory the graph assembler does not enumerate can never trip it. An undercount is invisible by construction.

That is the ten-month defect again: a universe smaller than the thing it measures, with nothing pointed at the difference — now sitting inside the instrument built to find exactly that. There is no fix, so the programme’s answer is that a known gap in the perimeter belongs in the ledger as a written English fact rather than inside the number, where it would be silently absorbed. Three uncounted readers are recorded that way today. The weakest link in the apparatus is a paragraph, which is at least the correct kind of weak.

There is a third failure mode, and it is the one I would bet on if I stopped watching: building the framework instead of fixing the pipeline. The charter forbids it in advance — every unit of work converts a seam or is declared a failure in the ledger with that word, explicitly including work that grows the graph, extends the assembler, or amends the charter itself. The clause exists because the foreseeable end state of a programme like this is arriving in October with a beautiful generated artefact, an armed ratchet, four reports, and the same three silent seams. That repository once counted its own written-down rules as progress and watched the number go 27, 51, 88, 115, 153, 169 across six commits, every step a widening, never once a shrink, on a mechanism whose documentation says shrinking is free. Prose accumulates; that is what prose does. Any metric that rises when you write things is measuring the writing.

So the answer to the question the last essay closed on — is this test still capable of telling me I am wrong? — is not a better test. The question is empirical. It has to be re-run against every guard at every boundary, and only an adversary with no stake in the answer being yes can settle it. A coding agent is unusually well suited to that, mostly because the job is to generate a great many wrong things quickly and care about none of them.

And the answer expires. A guard that killed its seed last month is telling you about last month, which is why the killing belongs in the build, where the claim either renews itself or goes red. That is the same trick as freezing the oracle before the code exists, and it has the same limit: it constrains what the machinery can do, and it cannot tell you what the machinery was never pointed at. Something has to keep asking that, and the only thing I have found that reliably does is a session that has never seen the reasons.