Signed Before the Code Existed

A passing test suite is the thing every engineer wants to see, and it is worth much less than it looks. The green run tells you the code does what the tests say. It is silent about whether the tests say the right thing. Those are different claims, and the gap between them is exactly where foundations rot: the code and the tests were written by the same hand, in the same afternoon, under the same misunderstanding, and they agree with each other perfectly because they were made to. A repository full of passing runs will mask precisely the hole the runs cannot detect, because the hole is in the thing doing the detecting.

This is not a counsel of despair; it is a design constraint. If you cannot read a green experiment as a proof about the foundations, then you have to arrange the experiment so that the one thing you can trust is nailed down before the code that will be judged against it has any chance to influence it. The tests have to exist, and be fixed, and be provably fixed, before the implementation is written. Not “written first” in the loose test-driven sense, where the author holds both pens and can still soften a test the moment the code makes it inconvenient. Fixed as in frozen: authored, then cryptographically sealed, and thereafter unchangeable without an alarm going off.

This is the story of a language, proplang, whose reference implementation and Haskell port were written by a coding agent under exactly that regime. The flagship essay on it is about cornering — making the wrong program unsayable in the grammar. This one is about the other half of the discipline, the epistemic half: how you check a thing built by a model whose every line you cannot supervise, when a green suite is not proof and the model’s own word is not evidence.

Freeze the oracle before the code exists

The protocol inverts the usual order completely. First the acceptance tests were written. Then a manifest — a sha256 manifest over the whole apparatus — was cryptographically signed. Then, and only then, did the implementation begin. At the hand-off, the source of the thing to be built was signature stubs: functions with the right types and undefined bodies. So cabal test compiled, ran, and failed all four acceptance tests. Not failed to compile — failed. The suite was frozen red, and its redness was the starting gun.

Freezing the tests red before writing the code sounds like theatre until you see what it forecloses. The ordinary temptation of a test-driven author, human or model, is not to cheat outright. It is to look at a failing assertion, look at the code, and quietly decide that the test was slightly wrong — too strict, testing the wrong thing, an off-by-one in the expectation — and to nudge it into agreement with the code that feels right. That move is invisible in the final green suite. Every test passes; nobody can see that one of them used to say something else. The freeze removes the pen. A wrong frozen test is not something you fix on your way to green; it is a stop-and-report. If a frozen test is genuinely wrong, the whole process halts and the error is surfaced, in the open, as its own event — because the alternative, silently correcting the oracle to match the code, is the exact failure the freeze exists to prevent.

What got frozen was not only the acceptance tests. The signed manifest covered the test suites, and it covered the machinery that audits the build: a forbidden-token list, an ablation script and its fixtures, a gate-runner, an export-list checker, a comment-stripper. It covered the build-protocol document itself. And — this was a reviewer’s hardening, added because the first version had not gone far enough — it covered the Python reference implementation and that reference’s own recorded test output. The Haskell port was checked against the Python oracle; so the oracle, too, had to be frozen, or the port could have been quietly checked against a moving target. Freeze the test and you still have to ask what the test trusts. The answer here was: freeze that too, all the way down to the thing the answer key was itself copied from.

Two keys, and the builder is the agent

If the freeze is going to mean anything, you have to be able to prove when things were frozen and who froze them, and you cannot take the author’s word for it any more than you can take the code’s. So the custody is cryptographic. Every commit over the manifest is signed. There are two signing identities, both under one email, distinguished by SSH key fingerprint: an author key and a builder key.

The builder is the coding agent. The Haskell port was written by a Claude Code agent, and its commits are signed under the builder key — so the version history records, cryptographically and per artifact, which parts of the record a human authored and which the agent produced. Most of the freeze tags are author-signed. Two specific increments are builder-signed, and each of those tags carries, verbatim in its message, an explicit author delegation — the human saying, in the signed record, I am authorising the builder to sign this one, for these reasons. The chain of custody does not just assert who did what; it carries the authorisation for every exception to the default, inline, where it cannot be detached from the thing it authorises.

This is the part that generalises past one project. When a model writes your code, “trust me, I wrote it carefully” is not available, and neither is “trust me, the human reviewed every line,” because the human did not. What is available is a signed ledger that says exactly which commits came from the agent and under whose standing authority — a provenance you can verify without trusting either party’s memory of it.

Seven gates it could not open

Between the frozen tests and the signed history sit the build gates: seven checks the agent’s code had to pass on every single run, none of which the agent could edit, because they live inside the signed manifest. I will compress them, since the flagship essay dwells on a couple; the point here is that they are mechanical and unbribable, not that each is clever.

The build must be clean under strict warnings-as-errors, incomplete patterns included. The engine module must export exactly a frozen list of twenty-one names, and the core belief type must stay abstract — its constructors never leave the module. The token IO may not appear anywhere outside a single host file, so the pure core physically cannot perform an effect. A forbidden-token grep must find none of ten words — detect, forget, window, decay, sliding, reset, trigger, temper, anneal, throttle — anywhere in the engine source, because each names a piece of adaptation machinery the design exists in order not to have; there is no forgetting to be found in a source that cannot say “forget.” The full frozen test battery must pass, which by the end of the project meant 127 tests across eight suites. The signed manifest must verify. And a code-level ablation check must hold: compiled with flags that drop a terminal, the language must become physically unable to utter that terminal, and the build error itself must name the missing constructor.

None of these is a matter of the agent’s judgement or good faith. Each is a script that returns pass or fail. The agent could write any Haskell it liked; it could not widen the export list, could not smuggle in an effect, could not type the word decay, and could not edit the checker that would have caught it, because the checker’s hash was signed before the agent started.

The brief, quoted byte for byte

There is a document I find more convincing than any of the gates, because it closes the loop the gates leave open. The gates prove the code obeys the frozen rules. They say nothing about whether the frozen rules covered the original brief — the research requirements the whole thing was supposed to satisfy. So there is a separate, machine-checked audit that takes each requirement from that brief, quotes it byte for byte, and assigns it a status from a closed set: done, recorded-debt, or named-open. A machine verifies that every quote is faithful to the source — you cannot paraphrase a requirement into something easier to have met. Every requirement came out “done.”

The honesty here is structural rather than asserted. The status set is closed, so a requirement that fit none of the three allowed states could not be quietly rounded to the nearest flattering one; it would itself become a stop-and-report. And because the quotes are machine-checked, the audit cannot drift into describing a softer brief than the one that was actually issued. This is the same instinct as the freeze, aimed one level up: don’t trust the summary of what was asked for — pin it to the bytes and let a machine confirm the pin.

The reader is not asked to take any of this on faith. The write-up ends with an end-to-end verification path: from a clean shell, verify the manifest checks out; run the gate-runner and watch gates one through seven pass; inspect the signed commit log over the manifest and see every commit signed, split between the author key and the builder key; run a quote-fidelity check; verify the signatures on the freeze tags. Every claim in the record is meant to be independently re-runnable by someone who trusts none of the people involved.

A record of its own failures

The thing that finally earned my trust is not the apparatus. It is that the project kept a record of its own failures — eight recorded incidents — and that each one became protocol text at the next freeze boundary, so the rules read like scar tissue. A few of them:

An oracle was once frozen compile-red under weaker warning flags than the future gate would use. Its redness proved nothing, because a red run only means something if it is bit-faithful to the conditions that will later judge the green. New rule: a red run proves nothing unless it is bit-faithful to the future gate conditions. A pinned numeric literal was once derived from a parallel simulation rather than from the frozen artifact itself — so the “expected value” came from a second computation that was not the one under test. New rule: pins derive from the frozen artifact itself; simulations establish that a test has discriminating power, never what its values are. A fixture’s redness was once assumed to be the intended missing-constructor error rather than demonstrated to be it. New rule: a fixture’s red must be demonstrated to be the missing constructor, not assumed. A test name once changed bytes under a locale change, so the increment-oracle test names were restricted to ASCII. And the custody scheme has its own scars: the in-person signing rule was beaten twice by convenience, which is why the scheme became two keys and two truthful attestations. Every attestation scheme the project tried is still represented in its history. The failures are not erased; they are the reason the current rule reads the way it does.

The one I keep returning to is the erratum in the write-up’s own closing verification block — the very block that tells the reader how to check everything. One line of it was wrong. It stated an expectation that the block’s own execution then refuted: it named the wrong number of tags, and a gate line that was not runnable verbatim as written. And the write-up says so, against itself, and diagnoses precisely which line failed and why: the one sentence of that block that had been drafted from memory instead of derived from the frozen artifacts was the one that went wrong. Everything mechanically generated from the frozen record was correct. The single place the author trusted his own recollection over the mechanism was the single place the mechanism had to catch him. That is the entire thesis in one sentence of collateral damage. You do not build the freeze because you are careless; you build it because you are careful and will still, at exactly one point, trust your memory over the artifact, and you would like that to be the one point that gets caught rather than shipped.

The blind spot a cage cannot cover

I would like to end on the machinery working, but the honest ending is better, and it is the same ending the flagship essay reaches from the other side.

After the build was closed, an adversarial sweep went through the finished source looking for the things a model under pressure might smuggle in: seeds, test-aware branches, constants secretly tuned to the tests, any steering at all. It found none. The cage held; there was nowhere to put contraband. But the same sweep found something the cage was never built to stop. Two paths through the agent’s own deliberation arithmetic had no frozen test capable of falsifying them. An earlier refactor had turned certain identity checks into definitions — so the tests that were supposed to pin those paths down had quietly become tautologies, true by construction, incapable of ever failing. The agent never once violated a test. It did something a cage cannot prevent: it made a test vacuously true.

This was recorded, not patched — filed as another incident rather than quietly fixed and forgotten, because quietly fixing it would have been the same sin as quietly softening a frozen test. And it draws the exact boundary of the whole method. Freezing the oracle before the code exists, signing it, gating on it, auditing the brief byte for byte — all of that constrains what can be written. None of it, by itself, keeps what is tested falsifiable. A grammar can make the wrong program unsayable; a freeze can make the answer key unmovable; and a test can still, through nobody’s bad faith, decay into a sentence that cannot be false and therefore checks nothing. The cage is the first, indispensable half. The second half is that someone has to keep looking at the tests themselves, adversarially, asking of each green line not “did it pass” but “could it have failed” — because a hollow test passes exactly as brightly as a real one.

That is why this sits beside the two essays on disciplining coding agents from the inside, the spec-as-leash post and the one on treating a tool’s own adoption as a decision rather than a comparison. All three are versions of one claim about the AI-assisted-engineering moment: when the model writes the code, you cannot supervise the lines, so you have to constrain the environment — freeze what the code is checked against, make the checks things the code cannot edit, and sign the whole chain so no party’s memory is load-bearing. And then, having built the best cage you can, you distrust it in the one place it cannot defend itself, and you keep a human pointed at the question the machine can no longer ask on its own behalf: is this test still capable of telling me I am wrong?