Loomworks — standing note: a check is not trusted until it has been observed failing — v0.4

Version. 0.4 Date. 2026-08-09 Supersedes. v0.3 at record b2b1ed8. All earlier versions stand as siblings. Changes from v0.3. A fourth pattern, from CR-2026-191: a test whose passing is the alarm. The three patterns before it are all about a green result that should not be trusted. This one is about a green result that should be investigated — where the natural reading of green is backwards.

Changes from v0.2, carried. A third failure mode is added, from CR-2026-190: a check that is sound but incomplete, and whose blind spot is filled in by a defect it does not cover — so the check reads clean while the thing it could not see stays hidden. The first two modes are about checks that cannot fail; this one is about a check that can fail, does fail, and is answered by fixing the wrong thing. Author. Claude Code (CR-2026-182 execution session). Operator: Marvin Percival. Charter. standing-notes/dunin7-standing-authorization-charter-v0_1. Origin. Five instances in the CR-2026-175→184 sequence of a check that reported green while unable to catch anything — or that could catch something but would not have said what. Reads with. standing-notes/loomworks-standing-note-safety-mechanisms-that-blind-v0_1; standing-notes/loomworks-standing-note-spend-context-built-at-the-gate-v0_1.

The rule

Any newly written check — test, guard, gate, assertion, lint rule — is not trusted until it has been observed failing for the reason it exists, in a way that names what broke.

Two requirements, and they are separable:

1. It must be observed failing. Deliberately, for its own reason, then observed passing again once the thing is restored. This proves the check can fail.
2. It must fail in a way that names what broke. This is a property of the assertion's content, not of the break-and-observe. A test that goes red without saying why sends the next person hunting, and it will usually be someone with less context than whoever wrote it.

v0.1 named only the first. The second is what makes a red test useful rather than merely honest.

This already applied to CI gates by practice (CR-2026-167 broke every gate before trusting it; CR-2026-173 and CR-2026-174 proved the mypy and ruff gates red-then-green on real infrastructure). It is now general. The sequence below shows the failure is not specific to CI.

Why: five instances in one sequence

| # | The check | Why it could not catch anything | Caught by | |---|---|---|---| | 1 | The route-dependency guard (CR-2026-178) | A request-based version returns 200 for a route that 422s every real caller — conftest stubs the very dependency that breaks it | break-and-observe | | 2 | The mypy baseline (CR-2026-181) | Four already-fixed errors still authorized; the floor had drifted looser than reality, so a regression reintroducing any of them would have passed | reading the regeneration diff | | 3 | The non-erasure guard (CR-2026-181) | Not vacuous — it fired correctly. The temptation was to exempt it for a variable named truncated, which would have made it vacuous for every future case | refusing the exemption | | 4 | The selector-mounting test (CR-2026-182) | Passed twice while incapable of failing: first the harness had no workspaces so a different guard suppressed the control anyway; then it rendered a bare <div> and never rendered the component under test | break-and-observe |

Break-and-observe caught three of the four. The one it did not — the drifted baseline — was caught by reading a diff, and is the reason the regeneration rule exists (scripts/check_mypy_baseline.py). That is worth stating plainly: break-and-observe is necessary and not sufficient. A check can also rot after being correctly proven, and nothing about proving it once prevents that.

A fifth instance, and it separates the two disciplines

CR-2026-184. Proving a new test could fail required reverting the fix — a filter widened in two places, a list query and its count query. The restore put back only one of the two. The count stayed narrow.

The test caught it, and not because of break-and-observe — that had already served its purpose by proving the test could go red. It was caught by what the assertion said:

> assert body["total_count"] == 1, "the count must agree with the list — a failed job counted in one and not the other is the same false impression in miniature"

A bare assert body["total_count"] == 1 would have failed identically and communicated nothing; the reader would have started debugging the count from scratch. The assertion's specificity is what made the failure informative rather than merely red.

Without that assertion the change would have shipped with a list and a count that disagreed — the exact defect the CR existed to remove, reintroduced in miniature by the act of proving the fix worked.

> The lesson, stated separably. Break-and-observe proves a check can fail. The content of its assertion decides whether that failure tells you anything. Both are required, and doing the first well says nothing about the second.

A third mode: a sound check, masked by a defect it does not cover

The two modes above are both about checks that cannot fail. CR-2026-190 found a third, and it is harder to see because the check works.

The check. CR-2026-190 applied a proposed NOT NULL constraint to the test database and ran the suite, to answer one question: is any failure a production path rather than a fixture? 226 tests failed, every one of them the same violation. Each failing test was traced, every production construction site was read directly, and the answer came back no — all fixtures. That answer was correct.

What it missed. Several of the failing tests exercised rebuild-from-scratch, which is production code. They failed for a fixture reason — their fixtures carried no number — so they were classified as fixtures and the fixtures were fixed. Fixing them made those tests green, and in doing so removed the only signal that the production path underneath them was also broken.

What survived was a single test whose fixture was not a defect: it modelled a genuine pre-Phase-36 payload, faithfully, and there was nothing in it to fix. That one test was the entire remaining evidence for a defect affecting nearly every historical event in the system.

> The shape of it. The check keyed on which tests failed. The rebuild tests failed for a fixture reason. Fixing those fixtures masked the real problem behind them. A defect the check did not cover was sitting behind a defect it did — and repairing the outer one closed the window on the inner one.

Why this is not the earlier modes. The check could fail, did fail, and failed loudly — 226 times. Its assertions named the invariant precisely. It was sound and incomplete, and incompleteness of that kind does not announce itself: the check goes quiet exactly when you have finished responding to it.

How to apply

• When a check reports a population of failures, ask what the population would look like if there were two causes. "All 226 are the same violation" is evidence of a shared symptom, not of a shared cause.
• Be suspicious of fixing your way to green through a large uniform failure set. Each fix removes a data point. If the classification was ever wrong, the evidence disappears as the work proceeds — and it disappears fastest at the start, while confidence is highest.
• Give special weight to the last few survivors. The failures that remain after the obvious class is cleared are not stragglers; they are the ones your model of the problem does not explain.
• Ask whether the check's question was the same as the question you cared about. "Is any failing test a production path?" and "does any production path write NULL?" are different questions. The first is a proxy, and proxies fail quietly at the edges.

A fourth pattern: a test whose passing is the alarm

The patterns so far all describe a check whose green is not evidence — the vacuous test that cannot fail, the baseline that drifted looser than reality, the sound check masked by a defect it does not cover. In each, green means nothing.

CR-2026-191 produced one where green means something, and the something is bad.

The test asserts that rebuilding a projection from a log that does not carry display_number returns nulls. It pins the defect B-70 existed to close. It passes today, and it is supposed to pass — that is the defect being present in the fixture's constructed conditions, which is what makes the adjacent test meaningful.

But if it ever passes in a world where the backfill has run, that is not the problem going away. It would mean the projector had started deriving numbers from somewhere other than the log — which is precisely the remedy the Operator ruled out permanently, because it makes a reader into a writer of facts and lets two rebuilds disagree.

> The natural reading of a green test is "the thing I care about is fine." For a defect pin, the natural reading is backwards, and it will be read by someone who does not know that.

How to apply

• Say so in the test itself. The docstring must state what a pass would mean under changed circumstances, because a passing test is never investigated and its name is all most readers see.
• Name it for the state, not the outcome. test_rebuild_loses_numbers_when_the_log_does_not_carry_them describes the world it pins. A name like test_rebuild_handles_missing_numbers would read as reassurance for the same assertion.
• Prefer pinning a defect to deleting the evidence of it. When a repair lands, the temptation is to remove the test that described the broken state. Keeping it is what makes a silent regression in the fix visible — but only if the reader knows which direction is bad.
• Distinguish it from a regression test out loud. A regression test says this must not come back. A defect pin says this is still true, and if it stops being true, ask why before celebrating. They look identical in a test report.

What makes this failure mode different

A vacuous check is indistinguishable from a passing one by reading the result. Every other defect class announces itself somewhere — a red test, an error, a wrong value on a screen. This one produces the exact output of success, and produces it reliably, which is worse than producing nothing: it is positive evidence for a claim nobody checked.

Instance 4 is the sharpest illustration. Two consecutive versions of the same test passed. Both looked like coverage. Neither could fail. The second was more elaborate than the first — mocking workspaces to remove a confound — and still tested nothing, because it never rendered the component. Effort spent on a check is not evidence the check works.

How to apply

1. Break the thing the check exists to catch, and watch the check fail. Not a nearby thing — the actual thing. If the check guards "route X is reachable", make route X unreachable.
2. Read the failure message, and make it worth reading. It must name the real reason. A check that fails with an incidental TypeError where an assertion was expected is telling you it fired for the wrong cause — that happened in CR-2026-176 and the assertions were rewritten before committing.

Write the message for someone who does not know why the check exists. assert x == 1 and assert x == 1, "the count must agree with the list" fail on the same input; only one of them ends the investigation at the point of failure. State what invariant broke, not what value was unexpected — the value is already in the traceback.

1. Restore, and watch it pass. Half the proof is that it goes green again for the right reason; a check that fails permanently is not evidence either.
2. For checks over a whole surface, assert non-vacuity in the check itself. The route guard asserts it inspected >100 routes, so it cannot silently become a no-op if route collection changes. A count assertion costs one line and survives refactors that quietly empty the input.
3. When a guard is inconvenient, move your code, not the guard. Instance 3: a local variable was renamed rather than the non-erasure guard exempted. An exemption is permanent and applies to every future case; a rename costs nothing and applies to one.
4. Ratcheting checks need a re-proof discipline, not only a first proof. A baseline, allowlist, or snapshot can loosen after it was correctly established. Whatever regenerates it must be read, not run and committed.

The general form

> Treat a green check as a claim, not as evidence — until you have seen it go red on purpose, and read what it said when it did.

The three standing notes now point at one thing from different angles: a mechanism can be correct and still hide what it was meant to reveal (spend context at the gate), a mechanism that makes something safe in one dimension can blind you in another (test isolation, unit coverage), and — here — a mechanism can report success while structurally incapable of reporting anything else. In all three, the artifact that looked like assurance was produced by the same machinery that suppressed the warning.

DUNIN7 — Done In Seven LLC — Miami, Florida Loomworks — standing note: observe the failure before trusting the check — v0.4 — 2026-08-09