Version. 0.6 Date. 2026-08-10 Supersedes. v0.5. All earlier versions stand as siblings. Changes from v0.5. The note's scope is corrected. Every instance in it until now was an automated check — and the pattern is not about automation at all. B-51's live verification produced a vacuous manual measurement, by the identical mechanism, with nothing to grep for and no suite to catch it. §"When the check is a person looking" is new.
Changes from v0.4, carried. A worked example of vacuity arriving through an improvement, from CR-2026-192. Every prior instance in this note came from a check written carelessly or left to rot. This one was introduced while deliberately making a test stricter, which is the version worth being able to recognise.
Changes from v0.3, carried. A fourth pattern, from CR-2026-191: a test whose passing is the alarm. The three patterns before it are all about a green result that should not be trusted. This one is about a green result that should be investigated — where the natural reading of green is backwards.
Changes from v0.2, carried. A third failure mode is added, from CR-2026-190: a check that is sound but incomplete, and whose blind spot is filled in by a defect it does not cover — so the check reads clean while the thing it could not see stays hidden. The first two modes are about checks that cannot fail; this one is about a check that can fail, does fail, and is answered by fixing the wrong thing.
Author. Claude Code (CR-2026-182 execution session). Operator: Marvin Percival.
Charter. standing-notes/dunin7-standing-authorization-charter-v0_1.
Origin. Five instances in the CR-2026-175→184 sequence of a check that reported green while unable to catch anything — or that could catch something but would not have said what.
Reads with. standing-notes/loomworks-standing-note-safety-mechanisms-that-blind-v0_1; standing-notes/loomworks-standing-note-spend-context-built-at-the-gate-v0_1.
Any newly written check — test, guard, gate, assertion, lint rule — is not trusted until it has been observed failing for the reason it exists, in a way that names what broke.
Two requirements, and they are separable:
v0.1 named only the first. The second is what makes a red test useful rather than merely honest.
This already applied to CI gates by practice (CR-2026-167 broke every gate before trusting it; CR-2026-173 and CR-2026-174 proved the mypy and ruff gates red-then-green on real infrastructure). It is now general. The sequence below shows the failure is not specific to CI.
| # | The check | Why it could not catch anything | Caught by |
|---|---|---|---|
| 1 | The route-dependency guard (CR-2026-178) | A request-based version returns 200 for a route that 422s every real caller — conftest stubs the very dependency that breaks it | break-and-observe |
| 2 | The mypy baseline (CR-2026-181) | Four already-fixed errors still authorized; the floor had drifted looser than reality, so a regression reintroducing any of them would have passed | reading the regeneration diff |
| 3 | The non-erasure guard (CR-2026-181) | Not vacuous — it fired correctly. The temptation was to exempt it for a variable named truncated, which would have made it vacuous for every future case | refusing the exemption |
| 4 | The selector-mounting test (CR-2026-182) | Passed twice while incapable of failing: first the harness had no workspaces so a different guard suppressed the control anyway; then it rendered a bare <div> and never rendered the component under test | break-and-observe |
Break-and-observe caught three of the four. The one it did not — the drifted baseline — was caught by reading a diff, and is the reason the regeneration rule exists (scripts/check_mypy_baseline.py). That is worth stating plainly: break-and-observe is necessary and not sufficient. A check can also rot after being correctly proven, and nothing about proving it once prevents that.
CR-2026-184. Proving a new test could fail required reverting the fix — a filter widened in two places, a list query and its count query. The restore put back only one of the two. The count stayed narrow.
The test caught it, and not because of break-and-observe — that had already served its purpose by proving the test could go red. It was caught by what the assertion said:
> assert body["total_count"] == 1, "the count must agree with the list — a failed job counted in one and not the other is the same false impression in miniature"
A bare assert body["total_count"] == 1 would have failed identically and communicated nothing; the reader would have started debugging the count from scratch. The assertion's specificity is what made the failure informative rather than merely red.
Without that assertion the change would have shipped with a list and a count that disagreed — the exact defect the CR existed to remove, reintroduced in miniature by the act of proving the fix worked.
> The lesson, stated separably. Break-and-observe proves a check can fail. The content of its assertion decides whether that failure tells you anything. Both are required, and doing the first well says nothing about the second.
The two modes above are both about checks that cannot fail. CR-2026-190 found a third, and it is harder to see because the check works.
The check. CR-2026-190 applied a proposed NOT NULL constraint to the test database and ran the suite, to answer one question: is any failure a production path rather than a fixture? 226 tests failed, every one of them the same violation. Each failing test was traced, every production construction site was read directly, and the answer came back no — all fixtures. That answer was correct.
What it missed. Several of the failing tests exercised rebuild-from-scratch, which is production code. They failed for a fixture reason — their fixtures carried no number — so they were classified as fixtures and the fixtures were fixed. Fixing them made those tests green, and in doing so removed the only signal that the production path underneath them was also broken.
What survived was a single test whose fixture was not a defect: it modelled a genuine pre-Phase-36 payload, faithfully, and there was nothing in it to fix. That one test was the entire remaining evidence for a defect affecting nearly every historical event in the system.
> The shape of it. The check keyed on which tests failed. The rebuild tests failed for a fixture reason. Fixing those fixtures masked the real problem behind them. A defect the check did not cover was sitting behind a defect it did — and repairing the outer one closed the window on the inner one.
Why this is not the earlier modes. The check could fail, did fail, and failed loudly — 226 times. Its assertions named the invariant precisely. It was sound and incomplete, and incompleteness of that kind does not announce itself: the check goes quiet exactly when you have finished responding to it.
The patterns so far all describe a check whose green is not evidence — the vacuous test that cannot fail, the baseline that drifted looser than reality, the sound check masked by a defect it does not cover. In each, green means nothing.
CR-2026-191 produced one where green means something, and the something is bad.
The test asserts that rebuilding a projection from a log that does not carry display_number returns nulls. It pins the defect B-70 existed to close. It passes today, and it is supposed to pass — that is the defect being present in the fixture's constructed conditions, which is what makes the adjacent test meaningful.
But if it ever passes in a world where the backfill has run, that is not the problem going away. It would mean the projector had started deriving numbers from somewhere other than the log — which is precisely the remedy the Operator ruled out permanently, because it makes a reader into a writer of facts and lets two rebuilds disagree.
> The natural reading of a green test is "the thing I care about is fine." For a defect pin, the natural reading is backwards, and it will be read by someone who does not know that.
test_rebuild_loses_numbers_when_the_log_does_not_carry_them describes the world it pins. A name like test_rebuild_handles_missing_numbers would read as reassurance for the same assertion.The instances above all came from a check written carelessly or left to rot. This one came from making a test better, and that is why it is worth recording specifically.
The change. CR-2026-192's constraint turned a rebuild that previously returned nulls into one that raises. B-70's defect pin was reframed to match — from asserting the returned values were null, to asserting the rebuild fails:
with pytest.raises(IntegrityError):
await rebuild_shape_events_view(engagement_id=eid, db=db)
Strictly better on its face. It fails loudly rather than returning nulls someone has to notice, and it proves the same property: the projector reads the number from the log rather than inventing one.
And it was vacuous. pytest.raises(IntegrityError) passes on any integrity error. A foreign key, a unique index, an unrelated constraint added later, a fixture that stopped producing valid rows — all of them satisfy it, and none of them proves anything about display numbers. The test would have gone green while the property it exists to protect went unchecked.
The fix is one clause:
assert "display_number" in str(exc.value), (
"the rebuild failed, but not on the missing display number — "
"something else broke and this test is no longer proving the "
f"projector reads the number from the log. Actual: {exc.value}"
)
> The shape to recognise: pytest.raises(SomeBroadError) with no assertion about the message. The broader the exception class, the more ways there are to pass without testing anything. IntegrityError covers every constraint in the schema; Exception covers everything that can go wrong.
Nothing about it looked like a mistake while it was being made. It was a deliberate improvement, written by someone actively thinking about the test's rigour, in a change request whose entire subject was classifying tests carefully. The care was real and pointed somewhere else — at whether the assertion matched the new behaviour, not at whether the new assertion could distinguish anything.
Rot and carelessness are the failure modes people look for. This one hides inside an upgrade, and the upgrade's own justification is what makes it feel finished.
pytest.raises needs a claim about which failure**, not only that one occurred — a message match, an error code, an attribute.pytest.raises(Exception) proves the code did not succeed; that is rarely the thing under test.A vacuous check is indistinguishable from a passing one by reading the result. Every other defect class announces itself somewhere — a red test, an error, a wrong value on a screen. This one produces the exact output of success, and produces it reliably, which is worse than producing nothing: it is positive evidence for a claim nobody checked.
Instance 4 is the sharpest illustration. Two consecutive versions of the same test passed. Both looked like coverage. Neither could fail. The second was more elaborate than the first — mocking workspaces to remove a confound — and still tested nothing, because it never rendered the component. Effort spent on a check is not evidence the check works.
TypeError where an assertion was expected is telling you it fired for the wrong cause — that happened in CR-2026-176 and the assertions were rewritten before committing.
Write the message for someone who does not know why the check exists. assert x == 1 and assert x == 1, "the count must agree with the list" fail on the same input; only one of them ends the investigation at the point of failure. State what invariant broke, not what value was unexpected — the value is already in the traceback.
> Treat a green check as a claim, not as evidence — until you have seen it go red on purpose, and read what it said when it did.
The three standing notes now point at one thing from different angles: a mechanism can be correct and still hide what it was meant to reveal (spend context at the gate), a mechanism that makes something safe in one dimension can blind you in another (test isolation, unit coverage), and — here — a mechanism can report success while structurally incapable of reporting anything else. In all three, the artifact that looked like assurance was produced by the same machinery that suppressed the warning.
Every instance above is an automated check. That framing was too narrow, and B-51 showed why.
B-51 turned on a question source-reading could not settle: does scroll position survive when the lens subtree is torn down and rebuilt? The answer needed a live measurement, so one was taken — scroll the page, click "Load more", compare scrollTop before and after.
The first measurement read 0 before and 0 after. Delta zero. Scroll survived.
It proved nothing. The document was not scrollable at that viewport — scrollHeight equalled innerHeight, so scrollTop could only ever have been 0. The measurement could not have produced any other number. It is precisely the first failure mode in this note — a check whose input is empty, reporting success — arriving in a person's hands rather than a test file.
What makes this version harder to catch, and worth its own section:
What caught it was the same discipline the rest of this note is about, applied to a measurement instead of a test: before believing this reading, could it have come out differently? It could not. The remeasurement forced a genuinely scrolled state first — confirming scrollTop was non-zero before clicking — and only then was the delta meaningful. The real answer happened to be the same, which is exactly why the first reading was dangerous rather than merely useless.
> The rule, generalised past its original wording. Establish that the check can fail before trusting that it did not — and "check" includes a person looking at a number. For a measurement the question has a specific form: what value would this have shown if the thing I am testing for were happening? If the answer is the same one, the measurement is not evidence.
A practical form for live verification. Assert the precondition first, out loud, as its own step: the container is scrollable, the list has a further page, the element is focused. Each of those is a non-vacuity assertion, and in the B-51 remeasurement each one was checked before the thing being measured. That is the manual equivalent of asserting the collection is non-empty before asserting nothing in it matches.
DUNIN7 — Done In Seven LLC — Miami, Florida Loomworks — standing note: observe the failure before trusting the check — v0.6 — 2026-08-10