Loomworks — topic-filtered recall — scoping note — v0.1

Document: loomworks-topic-filtered-recall-scoping-note-v0_1 Version: v0.1 Status: scoping note — settles the shape of the queued §18.2 work (topic-filtered / relevance-ranked recall) far enough to commission a Step 0 inspection brief and a CR. Four Operator decisions named below. Date: 2026-07-28 Grounding: read-only inspection of engine main 1aac815 (router.py — _route_ask_about_past_input :1529, _route_recall_personal :1566, _PAST_INPUT_LIMIT :119, _PERSONAL_RECALL_LIMIT comment :121–127; prompt.py _format_past_input :481; classifier.py intent declarations :94–103); queued-directions v0.30 §18.2 (priority raised 2026-07-28 with V5 evidence); loomworks-vertical-vc-diligence-cc-inspection-report-v0_1 finding as carried in §18.2. Design inputs on file: loomworks-recordable-facts-and-topic-investigation-v0_1 (topic-carrying assertions); loomworks-conversation-search-scoping-note-v0_1 (message-stream search — adjacent, distinct, unchanged by this note).

Plain-language summary

Today, when you ask the Companion what is on record, it does not search. It grabs the 50 most recent committed notes and hopes the answer is among them. Your question never selects anything. On a small engagement this looks fine; on a real one, anything said early falls outside the window and the Companion silently fails to find it — the exact probe a skeptical prospect would run. No search index has ever existed in the engine. A standing conduct rule already blocks live recall demonstrations in front of prospects until this lands.

This note settles the fix in two tiers. Tier 1 (this build): a real text search — the question's terms select the matching notes from the complete committed record, using PostgreSQL's built-in full-text search with an index. Early content becomes findable by its words. No new infrastructure. Tier 2 (later, after the topic investigation settles): recall by meaning, so "what's the weather note" finds "it is sunny" — which word-matching cannot do.

What you decide: the four decisions in the last section — the tier split, which recall path builds first, what happens on zero matches, and whether Tier 1 relaxes the no-demo rule.

1. The evidenced failure (why this is now, not queued)

From the venture-diligence vertical inspection (V5, engine 1aac815), carried into queued-directions v0.30 §18.2:

• Recall hands the responder an unfiltered window of the 50 most recent committed assertions (_PAST_INPUT_LIMIT = 50, router.py:119). The Operator's question never selects anything.
• No search index has ever existed — zero hits repo-wide and across migration history for to_tsvector, tsquery, ILIKE, pg_trgm.
• The failure is silent: recall demonstrates well small and fails invisibly at real size. The natural skeptical probe — ask about something said early — is precisely the query that lands outside the window.
• Standing conduct rule in force: no live recall demonstration in front of a prospect until this lands. Product-wide; every vertical inherits it.

2. The two recall paths today (grounded)

| | ask_about_past_input (engagement) | recall_personal (personal, CR-2026-100) | |---|---|---| | Selection | 50 most recent committed (_PAST_INPUT_LIMIT) | Wholesale — complete personal set (_PERSONAL_RECALL_LIMIT, effectively-wholesale backstop) | | Who composes | Responder (LLM focuses on the question) | Server-composed faithful readout (CR-2026-129 — exact list, real count, overflow disclosed) | | Completeness posture | Source comment: a pagination cap, not a completeness contract | The CR's explicit "wholesale, no silent budget-clip" requirement | | Question-driven? | No | No |

CR-2026-129's tests deliberately left ask_about_past_input untouched — "carved out to the semantic-recall foundation" (test_cr_2026_129_recall_truthfulness.py:182–202). This note is that reserved work being picked up.

3. The two-tier shape

Tier 1 — deterministic full-text search (this build)

Replace the recency window with query-driven selection over the complete committed set:

• Index: PostgreSQL full-text search — a tsvector expression (or stored generated column) over assertion content with a GIN index. Native to the existing database; no external search service, no embeddings, no new dependency. Migration required (index; possibly a generated column) — migration discipline applies (single head; confirm current head at Step 0).
• Selection: extract search terms from the Operator's utterance at the route seam, run tsquery against committed assertions on the engagement, return matches ranked (ts_rank) — not the 50 newest. Early content becomes findable by its words regardless of age or record size.
• Honest-selection readout: the reply must present itself as matches for the question, never as the complete record — total committed count and matched count both surfaced, in the CR-2026-129 faithful-readout manner. A filtered subset silently posing as the whole record would recreate the silent failure in a new form.
• Cap semantics change meaning: any limit on returned matches becomes a display cap on an already-relevant ranked set, disclosed — not a silent relevance filter.

What Tier 1 fixes: the keyword-shaped probe ("what did I say about the vesting schedule?") finds the early note. What it does not fix: the paraphrase-shaped probe ("what's the weather note" → "it is sunny"). That is Tier 2's gap, and it is honest to say so.

Tier 2 — semantic / topic recall (after the topic investigation)

Recall by meaning rather than word overlap. Two candidate mechanisms, deliberately not chosen here: topic-carrying assertions (the recordable-facts investigation's direction — assertions carry a topic at commit time; recall matches on topic) and/or embedding search. The recordable-facts-and-topic investigation is the design input and must settle before any Tier 2 build — it touches Memory structure and the thesis. Tier 1 neither blocks nor prejudges it: a full-text index over content coexists with either mechanism.

4. Scope: engagement path first; personal path unchanged

Tier 1 targets _route_ask_about_past_input (the engagement path — where the evidenced failure lives and where record sizes are real). recall_personal stays wholesale: at personal-memory sizes the complete set is correct, its faithful readout already discloses overflow, and CR-2026-100's "wholesale is the foundation; any filter is a subset" framing holds. When personal size warrants, the same search path extends to it — a follow-on, not this build.

5. What this note deliberately does not do

• No seed reframe. §18.5 (memory = recall as the product) awaits explicit Operator commission. This build strengthens the case; it does not perform the reframe.
• No conversation search. The message stream is a different body; its scoping note stands unchanged, queued.
• No topic mechanism. Tier 2's design belongs to the recordable-facts investigation.
• No classifier change. Intent labels (ask_about_past_input, recall_personal) are untouched; this is retrieval, not routing.

6. Operator decisions

1. Tier split. Tier 1 now (deterministic full-text), Tier 2 after the topic investigation settles. Default: yes.
2. Path scope. Engagement recall builds first; personal recall stays wholesale. Default: yes.
3. Zero-match behavior. When the search matches nothing: say so honestly and offer the record ("No notes match 'X'. I hold N committed notes — want the recent ones / all of them?") — versus silently falling back to the recency window. Default: say so and offer; a silent fallback would blur what the answer is.
4. Conduct rule after Tier 1. Does Tier 1 relax "no live recall demonstration" to permit keyword-shaped probes (paraphrase-shaped probes still off-limits until Tier 2), or does the rule stay fully in force until Tier 2? Default: relax to keyword-shaped only, stated in the vertical documents as a scoped relaxation, not a removal.

7. Next steps after Operator decisions

1. Step 0 inspection brief for CC: confirm list_assertions signature and whether search pushes into it or a sibling query; confirm current alembic head for the index migration; confirm the term-extraction seam (route-level vs a small shared helper); confirm _format_past_input's rendering path for the honest-selection readout; report any existing ts_rank/FTS usage anywhere (expected: none).
2. CR drafting against the brief — engine-only expected (no frontend change; the readout is server/responder text).
3. Record: this note commits to loomworks-record/scoping-notes/; queued-directions §18.2 gains a cross-reference at its next bump.

DUNIN7 — Done In Seven LLC — Miami, Florida Loomworks — topic-filtered recall — scoping note — v0.1 — 2026-07-28 Tier 1: deterministic full-text search (tsvector + GIN) replaces the 50-recent window on engagement recall — query-driven selection over the complete committed set, honest-selection readout, migration-bearing. Tier 2: semantic/topic recall, gated on the recordable-facts-and-topic investigation. Personal recall stays wholesale. Four Operator decisions: tier split, path scope, zero-match behavior, conduct-rule relaxation.