Loomworks — standing note: a check is not trusted until it has been observed failing — v0.2

Version. 0.2 Date. 2026-08-07 Supersedes. v0.1 at record ace344b, which stands as a sibling. Changes from v0.1. The rule is split into two separable disciplines. v0.1 treated "observed failing" as the whole requirement; CR-2026-184 showed that being able to fail and failing informatively are different properties, and only the first was named. Author. Claude Code (CR-2026-182 execution session). Operator: Marvin Percival. Charter. standing-notes/dunin7-standing-authorization-charter-v0_1. Origin. Five instances in the CR-2026-175→184 sequence of a check that reported green while unable to catch anything — or that could catch something but would not have said what. Reads with. standing-notes/loomworks-standing-note-safety-mechanisms-that-blind-v0_1; standing-notes/loomworks-standing-note-spend-context-built-at-the-gate-v0_1.

The rule

Any newly written check — test, guard, gate, assertion, lint rule — is not trusted until it has been observed failing for the reason it exists, in a way that names what broke.

Two requirements, and they are separable:

1. It must be observed failing. Deliberately, for its own reason, then observed passing again once the thing is restored. This proves the check can fail.
2. It must fail in a way that names what broke. This is a property of the assertion's content, not of the break-and-observe. A test that goes red without saying why sends the next person hunting, and it will usually be someone with less context than whoever wrote it.

v0.1 named only the first. The second is what makes a red test useful rather than merely honest.

This already applied to CI gates by practice (CR-2026-167 broke every gate before trusting it; CR-2026-173 and CR-2026-174 proved the mypy and ruff gates red-then-green on real infrastructure). It is now general. The sequence below shows the failure is not specific to CI.

Why: five instances in one sequence

| # | The check | Why it could not catch anything | Caught by | |---|---|---|---| | 1 | The route-dependency guard (CR-2026-178) | A request-based version returns 200 for a route that 422s every real caller — conftest stubs the very dependency that breaks it | break-and-observe | | 2 | The mypy baseline (CR-2026-181) | Four already-fixed errors still authorized; the floor had drifted looser than reality, so a regression reintroducing any of them would have passed | reading the regeneration diff | | 3 | The non-erasure guard (CR-2026-181) | Not vacuous — it fired correctly. The temptation was to exempt it for a variable named truncated, which would have made it vacuous for every future case | refusing the exemption | | 4 | The selector-mounting test (CR-2026-182) | Passed twice while incapable of failing: first the harness had no workspaces so a different guard suppressed the control anyway; then it rendered a bare <div> and never rendered the component under test | break-and-observe |

Break-and-observe caught three of the four. The one it did not — the drifted baseline — was caught by reading a diff, and is the reason the regeneration rule exists (scripts/check_mypy_baseline.py). That is worth stating plainly: break-and-observe is necessary and not sufficient. A check can also rot after being correctly proven, and nothing about proving it once prevents that.

A fifth instance, and it separates the two disciplines

CR-2026-184. Proving a new test could fail required reverting the fix — a filter widened in two places, a list query and its count query. The restore put back only one of the two. The count stayed narrow.

The test caught it, and not because of break-and-observe — that had already served its purpose by proving the test could go red. It was caught by what the assertion said:

> assert body["total_count"] == 1, "the count must agree with the list — a failed job counted in one and not the other is the same false impression in miniature"

A bare assert body["total_count"] == 1 would have failed identically and communicated nothing; the reader would have started debugging the count from scratch. The assertion's specificity is what made the failure informative rather than merely red.

Without that assertion the change would have shipped with a list and a count that disagreed — the exact defect the CR existed to remove, reintroduced in miniature by the act of proving the fix worked.

> The lesson, stated separably. Break-and-observe proves a check can fail. The content of its assertion decides whether that failure tells you anything. Both are required, and doing the first well says nothing about the second.

What makes this failure mode different

A vacuous check is indistinguishable from a passing one by reading the result. Every other defect class announces itself somewhere — a red test, an error, a wrong value on a screen. This one produces the exact output of success, and produces it reliably, which is worse than producing nothing: it is positive evidence for a claim nobody checked.

Instance 4 is the sharpest illustration. Two consecutive versions of the same test passed. Both looked like coverage. Neither could fail. The second was more elaborate than the first — mocking workspaces to remove a confound — and still tested nothing, because it never rendered the component. Effort spent on a check is not evidence the check works.

How to apply

1. Break the thing the check exists to catch, and watch the check fail. Not a nearby thing — the actual thing. If the check guards "route X is reachable", make route X unreachable.
2. Read the failure message, and make it worth reading. It must name the real reason. A check that fails with an incidental TypeError where an assertion was expected is telling you it fired for the wrong cause — that happened in CR-2026-176 and the assertions were rewritten before committing.

Write the message for someone who does not know why the check exists. assert x == 1 and assert x == 1, "the count must agree with the list" fail on the same input; only one of them ends the investigation at the point of failure. State what invariant broke, not what value was unexpected — the value is already in the traceback.

1. Restore, and watch it pass. Half the proof is that it goes green again for the right reason; a check that fails permanently is not evidence either.
2. For checks over a whole surface, assert non-vacuity in the check itself. The route guard asserts it inspected >100 routes, so it cannot silently become a no-op if route collection changes. A count assertion costs one line and survives refactors that quietly empty the input.
3. When a guard is inconvenient, move your code, not the guard. Instance 3: a local variable was renamed rather than the non-erasure guard exempted. An exemption is permanent and applies to every future case; a rename costs nothing and applies to one.
4. Ratcheting checks need a re-proof discipline, not only a first proof. A baseline, allowlist, or snapshot can loosen after it was correctly established. Whatever regenerates it must be read, not run and committed.

The general form

> Treat a green check as a claim, not as evidence — until you have seen it go red on purpose, and read what it said when it did.

The three standing notes now point at one thing from different angles: a mechanism can be correct and still hide what it was meant to reveal (spend context at the gate), a mechanism that makes something safe in one dimension can blind you in another (test isolation, unit coverage), and — here — a mechanism can report success while structurally incapable of reporting anything else. In all three, the artifact that looked like assurance was produced by the same machinery that suppressed the warning.

DUNIN7 — Done In Seven LLC — Miami, Florida Loomworks — standing note: observe the failure before trusting the check — v0.2 — 2026-08-07