Document: loomworks-list-loading-cr-1-implementation-notes-v0_1
Status: complete — built, suite green, HALTED before push
Date: 2026-06-27
By: Claude Code (CC) on DUNIN7-M4
Implements: loomworks-list-loading-cr-1-engine-cursor-contract-v0_3
Repo: loomworks-engine, branch list-loading-cr-1-engine-cursor-contract
> Engine-only CR. Ten listing endpoints brought to one keyset contract, plus a shared codec (Step 0) and one migration (Step 0b: a function + two indexes). Pure-read GET paths; per-step commits; suite green at each step. Nothing pushed — awaiting Operator review.
| | |
|---|---|
| Final engine HEAD | 46d655e (branch list-loading-cr-1-engine-cursor-contract, off baseline 46ebaeb) |
| Commits | 13 (Step 0, Step 0b, Steps 1–10, + API-docs regen) — listed below |
| Suite | 3032 passed / 63 skipped / 0 failed (was 2973/63 at baseline; +59 = new keyset tests + codec units) |
| Alembic head | 0096 (was 0095) — one revision, additive + reversible |
| D-4 path taken | simple composite — playground_dev had 12 manifestations, zero null derived_at; null-rank machinery dropped |
| Wrapper function | lw_assertion_sort_key(payload jsonb, last_updated_at timestamptz) → timestamptz, IMMUTABLE, SET TimeZone='UTC' |
| Codec module | src/loomworks/pagination.py (relocated from api/pagination.py — see Deviation 1) |
| Planner check (playground_dev, enable_seqscan=off) | assertions: Bitmap Index Scan on ix_cmo_assertion_sort_key · shapings: Bitmap Index Scan on ix_cmo_shaping_created_at |
| Pure-read guard | held throughout — no GET handler writes / derives / touches a provenance thread |
The contract every endpoint now returns: limit + opaque cursor request params; has_more + total_count + next_cursor response fields; composite (sort_key, id) keyset boundary (row-value < for DESC/DESC, > for ASC/ASC, OR-expansion for mixed).
| Commit | Step | What |
|---|---|---|
| 669dc6c | 0 | Lift + generalise the cursor codec (type-agnostic + direction-aware); 4 Group A endpoints re-pointed, unchanged |
| e9aac38 | 0b | Migration 0096: lw_assertion_sort_key + ix_cmo_assertion_sort_key + ix_cmo_shaping_created_at |
| 8e4c3d4 | 1 | GET /engagements — sort e.id → (created_at DESC, id) |
| b9b2f5f | 2 | GET …/considerations (last_updated_at, object_id) + codec relocation to loomworks/pagination.py |
| c8577cb | 3 | GET …/compositions (created_at, object_id) — added the missing object_id tiebreak |
| 48aec73 | 4 | GET …/shape-events — INT cursor (engagement_version_at_production, object_id) ASC |
| dd54018 | 5 | GET …/assertion-groups — in-Python (group.created_at, group.id) over the snapshot |
| 6fb6568 | 6 | GET …/manifestations — (derived_at, object_id); D-4 simple composite |
| e3ebca8 | 7 | GET …/shapings — JSON-text ((payload->>'created_at'), object_id) on the index |
| 6fd6cc3 | 8 | GET …/assertions — wrapper keyset, mixed direction; offset removed |
| 35523ac | 9 | GET …/renders — (created_at, object_id) ASC preserved |
| a6cb020 | 10 | GET /operator/conversation-history — before → composite cursor; + total_count (Axis 11 R2 close) |
| 46d655e | — | Regenerate docs/api/* (keeps the api-docs freshness CI gate green) |
Per-step: explicit-path staging (never git add -A), suite green before the next step, lint (ruff F/B904/I001) clean on touched files.
src/loomworks/pagination.py: encode_cursor / decode_cursor / keyset_clause / cursor_params / next_cursor.
datetime encodes to the legacy {"ts":"<iso>","id":"<id>"} shape with no discriminator (Group A cursors byte-identical); int (shape-events) and str (shapings JSON-text) carry a "k" tag so decode restores the exact Python type.keyset_clause emits a row-value tuple comparison for same-direction columns (< DESC, > ASC) and an explicit OR-expansion for mixed direction (which a row-value comparison can't express — assertions needs value-DESC/id-ASC).needs_you/recent/active/inbox) page identically through the relocated codec — 38 Group A tests stayed green unchanged; 15 new codec unit tests lock the generalisations.0096)((payload->>'created_at') DESC, object_id ASC) partial WHERE object_type='shaping'. Directly IMMUTABLE.COALESCE((payload->>'created_at')::timestamptz, last_updated_at) sort is rejected by Postgres (the text→timestamptz cast is STABLE, not IMMUTABLE — verified empirically). Resolved per the Operator's fork decision with a UTC-pinned IMMUTABLE wrapper lw_assertion_sort_key, indexed (lw_assertion_sort_key(payload, last_updated_at) DESC, object_id ASC) partial WHERE object_type='assertion'.playground_dev). Both DBs (playground_dev, playground_test) at 0096.
1. Codec home — relocated to a neutral top-level module. The CR leaned src/loomworks/api/pagination.py but invited a better home. Six engagement-layer query modules consume the codec (considerations, assertions, shapings, manifestation, shape-events, renders); importing it from api/ would be a reverse layer dependency. I lifted it to src/loomworks/pagination.py (a pure leaf — only stdlib + fastapi). Done as part of Step 2; dashboard + engagement_list + the codec test re-pointed. No behaviour change.
2. Wrapper variant — SET TimeZone='UTC' + ::timestamptz, NOT the literal ::timestamp AT TIME ZONE 'UTC'. (Flagged for review.) The CR's parenthetical named AT TIME ZONE 'UTC'. But that form strips offsets — it interprets a parsed wall-clock as UTC, so a non-UTC value (+05:00) lands at its wall-clock-as-UTC moment, not its true instant — which would fail the very property the wrapper was chosen for over a text keyset ("a non-UTC value lands at its true moment"), and would NOT "mirror the live COALESCE semantics exactly" (the live code uses ::timestamptz, which respects offsets). I used a LANGUAGE sql IMMUTABLE SET TimeZone='UTC' function whose body is COALESCE((payload->>'created_at')::timestamptz, last_updated_at) — offset-respecting (verified: +05:00 → true instant), UTC-pinned, never-session-local, IMMUTABLE, indexable (all verified empirically). The SET TimeZone='UTC' clause IS the load-bearing UTC pin; the migration carries the comment. If you specifically wanted the offset-stripping AT TIME ZONE 'UTC' form, say so and I'll switch — it's a one-line change.
3. KeysetPage schema mixin. Added to api/schemas.py; most list responses now inherit total_count + has_more + next_cursor. (Endpoints with a pre-existing total_count — assertions, renders — got has_more+next_cursor added without the mixin to preserve their tailored field descriptions.)
4. has_more via fetch-limit+1 everywhere. Every endpoint over-fetches one row to decide has_more accurately, then trims — more accurate than the Group A "short page ⇒ no next" heuristic (which can return a spurious empty next page on an exact-multiple final page). next_cursor is non-null iff has_more.
5. Wholesale siblings kept where the function had other callers. considerations, shapings, assertions each gained a *_page sibling and left the original wholesale function untouched (it has ~20, 1-agent, and ~10 other callers respectively). The rest (engagements, compositions, shape-events, manifestations, renders, conversation-history) had a single caller and changed in place.
6. Assertions — offset removed (breaking response + param change). Per the CR ("replace offset with cursor"). AssertionListResponse drops offset, keeps limit + total_count, gains has_more + next_cursor. The prior offset-pagination test (test_phase_16…pagination) and an exact-dict assertion were rewritten to the cursor mechanism.
7. Step 5 known limitation (recorded follow-on). assertion-groups paginates in Python over the fully-materialised engagement snapshot — the cursor bounds the response payload, not server cost; the whole snapshot is built regardless. total_count is the honest full group count. Pushing sort/slice into the snapshot computation is a separate scoping question, named in the handler docstring and the CR §9 — out of CR-1.
8. Planner check methodology. playground_dev is tiny (112 assertions / 5 shapings), so the planner naturally prefers a seq scan regardless of index. To prove the index expression matches the query expression verbatim (the actual point of the check), I forced SET enable_seqscan=off and confirmed each index is then chosen on the keyset boundary (plan lines above). The two endpoints also carry an in-suite planner test asserting the same on playground_test.
9. D-7 unchanged here. conversation-history before→cursor is a breaking param change for its two OL callers (useConversation.ts, ChatView.tsx). Per the CR these are updated in CR-2's frontend pass, not here. The endpoint will 200 on a cursor arg and ignore an unknown before arg, so the old frontend would silently lose its "load earlier" paging until CR-2 lands — expected, since CR-2 is sequenced right after CR-1 merges.
Each of the ten has: first-page ≤ limit; has_more true iff a further page exists; next_cursor round-trips (page 2 starts exactly after page 1, no skip/repeat); a boundary-tie test (two+ rows sharing the sort value straddling the page boundary, both returned — the R2 guard); total_count = wholesale count under the same filters. Shapings + assertions additionally carry a planner check. New test files: test_pagination_codec.py + test_list_loading_{engagements,considerations,compositions,shape_events,assertion_groups,manifestations,shapings,assertions,renders}.py; conversation-history added to test_phase_46_conversation_history.py.
Seeding notes: base projection tables (render_compositions_view, shape_events_view, manifestation_view, render_events_view) seed directly; assertion-groups/shapings/assertions pin created_at via jsonb_set on memory_events/current_memory_objects; manifestations pin in both the view and the payload (the ORDER BY reads the view, the cursor reads the payload — they must agree); conversation-history pins created_at via UPDATE.
0096); reversibletotal_count matches the wholesale count under each endpoint's filtersplayground_devlist-loading-cr-1-engine-cursor-contract is local only.lists.page_size override, home-lens wiring, and the two conversation-history call-site updates (D-7) are all CR-2.HALT. Awaiting Operator review before any push.
DUNIN7 — Done In Seven LLC — Miami, Florida
Loomworks list-loading — CR-1 implementation notes — v0.1 — 2026-06-27
Engine branch list-loading-cr-1-engine-cursor-contract @ 46d655e; alembic 0096; 3032/63/0; not pushed.