Mindbridge · track status

FOLD-EFFICIENCY

The cross-source fold exceeds its 3h ceiling without document extraction's help — and the doc-needs precompute it feeds is 87% dead weight by rows.

Created 2026-08-26, spun out of the REGINA-DEDUP incident's fold-cost program (its parent cohort and the #3495 waves still live on that page) and founded on the measured exploration docs/exploration/20260826-regina-fold-cost-attribution.md — every founding figure there is [MEASURED], prod read-only. Counts resolve at build.

Next →🎯 #3601 — store top-K needs per document (~865K → ~316K rows), then #3602's HNSW-backed build. The completeness gap (#3600, see History) is closed — doc_extract_drain.py now arms a catch-up fold per tenant per batch via enqueue_cross_source, verified live in prod (30 work_queue rows, latest 2026-08-31). What remains on this track is the sizing pair.
24 open. The founding exploration answered a scoping question decisively: document extraction is not in the fold, has never been, and the fold exceeds the 3h ceiling without it — 5,717 Regina runs carry document_extraction in phase_results and zero also carry a fold phase; the 4.33h fold that died on 08-25 had no document-extraction entry at all. The cost is two phases — graph_cross_conv_matching (95.9 min) and graph_aboutness_densification (74.0 min), together 83.4% of phase time. The adjacent doc-needs precompute sizing program opened the same day: its ceiling fix (0.35 → 0.30) merged as PR #3594; the top-K store and the HNSW-backed build are the open rows below.
24open issues
203.6 mintotal phase time · 3h ceiling
83.4%two phases: matching + densification
87%doc_needs rows no caller can read
Open workWhy this orderStanding rulingsHistory

Open work

The doc-needs precompute — sized to what callers read
p2 #3601 Store top-K needs per document in library_doc_needs (~865K → ~316K rows) K≈20 sized from the measured rank distribution All

On top of the ceiling fix that merged as PR #3594 (6.65M → ~865K rows): keep the best K≈20 needs per document — the worst rank any caller ever reads is 7, so K=20 carries ~3× headroom. The reads are top-1-per-document (DISTINCT ON … ORDER BY distance); the store carries 453 needs per document to answer it. The row reduction is a direct Supabase IO/disk-charge cut on a table rebuilt every full pass.

p2 #3602 Build the precompute via HNSW top-K probes (~14× faster) with a node_type-partial index two silent traps measured — see the row All

Measured on Regina prod: 0.071 s vs 0.98 s per probe. 🪤 The two traps that make the obvious query WRONG: pgvector only uses the HNSW index for a constant/InitPlan probe — a joined-relation probe silently falls back to a full sort slower than brute force (8.66 s); and the index is tenant-partial but not node_type-partial, so the document filter discards every candidate and returns zero rows in 17 ms — fast, green, and wrong. The node_type-partial index is the safer fix and the natural builder for #3601's top-K store. Sequenced after it for that reason.

p2 ✅ PR #3594 Align the doc_need store ceiling to the read default (0.35 → 0.30) merged 2026-08-26 — 2,540 MB → ~330 MB All

🟢 Merged 2026-08-26. The sole caller never passes p_max_distance and the reader clamps to ≤0.35, so the 0.30–0.35 band was 5,789,296 rows (87%, ~2.2 GB) no caller could reach — pure dead weight rebuilt every full pass. Zero semantic change: the store ceiling now equals the read default.

Why is it in this order?

Why is it in this order?

Completeness gap closed; sizing is what remains. The enqueue gap (#3600) was silent data-not-picked-up — the same class as the incident that spawned this track — and shipped and verified live in prod 2026-08-31, so it no longer sets the order. What is left is the sizing pair: the top-K store (#3601) is independent and delivers the row reduction on its own; the HNSW build (#3602) is the accelerator for it and carries the index migration, so it lands second.

Deliberately ignored: the ~14× speed headline (a fast build of an oversized store is the wrong win); filing order; the REFUTED option 4 (drop the table, serve on demand — measured at ~7 minutes per request, the precompute is legitimately required).

What the audit found — generated

⚠︎ The prose above cites 1 issue that is no longer open

#3602. A closed issue named as current work is the drift this section exists to surface — re-derive the prose, or move the reference into History.

Evidence below is generated from the read-only audit of 2026-08-29 and reconciled against the open issues at build time. Every claim cites the issue it came from. Nothing here is a decision — rulings stay in the prose above, which is written by hand.

#3495 [fold-efficiency] the fold's dominant cost is graph_cross_conv_matching +…commithigh confidence

Only the DOCUMENT leg was scoped to delta: commit 900ef0436 on origin/main ([fold-efficiency] fix(backend,db): scope fold document reads to per-conversation delta with an independent sync watermark (#3495) (#3521)). Its own commit message states 'PersonMaterializer is intentionally left untouched ... Flagged as owner-gated follow-up, not solved here.' No commit on origin/main since 2026-08-18 addresses the typed-item leg (git log origin/main --since=2026-08-18 -i --grep 3495 returns exactly two commits: 900ef0436 and dc71c85b1, the latter being CI wiring for #3502). Prod corroboration (public.ingestion_runs, last 5 days, grouped by tenant/day): the cost is still fleet-wide and heavy — 2026-08-28 tenant e9d96b09 (Regina) 637 runs / 51 failed / avg 1,724s / max 33,117s; tenant be0f3db0 178 runs / 18 failed / avg 559s / max 25,944s; 2026-08-27 avg 2,620s Regina and 2,280s on tenant 23d86e0d. Caveat: ingestion_runs durations are whole-run, not the graph_materialization substep alone.

#3531 The ADR-111 skip-check digest is itself O(tenant) — a 5-way full-tenant re-aggregate runs…codehigh confidence

origin/main:backend/app/slices/graph_building/repositories/node_index_repository.py:106-145 — `_PERSON_ATTRIBUTE_DIGEST_SQL` still runs five independent full-tenant aggregates (person_valuable_message_counts, person_impersonation_verdicts, person_automated_verdicts, whatsapp_jid_mappings, person_identities), each `string_agg`+`md5` with no watermark/delta predicate. Reachable on every fold: `graph_materializer.py:409` and `materialize_scan_planner_service.py:141` both call `person_attribute_input_digest()` unconditionally. `git log origin/main -- node_index_repository.py` shows the newest touch is #3355 (2026-07, WIRE-TRUTH), nothing after the 2026-08-22 filing.

#3532 graph_cross_conv_matching costs more than the entire fold on healthy tenants (propose…prodhigh confidence

Query on prod `public.ingestion_runs` (phase_times JSON), runs since 2026-08-22: cross-conv vs fold seconds — Regina e9d96b09: 5403 vs 726 (08-29 08:06Z), 4622 vs 1445 (08-29 04:09Z), 5694 vs 372 (08-28 23:11Z), 5315 vs 2038 (08-28 17:39Z); contrast tenant 23d86e0d: 128 vs 41, 109 vs 40, 97 vs 30. The premise holds and has WORSENED: on Regina cross-conv is 3-15x the fold and is measured in thousands of seconds. Code side: `git log origin/main --date=short -- backend/app/slices/graph_building/phases/graph_cross_conv_matching_phase.py` shows only #3594 (doc_need ceiling) and #3526 since filing — no diagnosis or perf change to propose/judge/write_edges.

#3601 Store top-K needs per document in library_doc_needs (~865K to ~316K rows on top of the…prodhigh confidence

Installed writer in prod carries NO top-K: select pg_get_functiondef(oid) like '%LIMIT%' for public.refresh_library_doc_need_matches_for_docs -> false; the body is the unbounded INSERT ... SELECT from nodes_v2 d JOIN nodes_v2 nd WHERE distance < 0.30 (origin/main:supabase/migrations/20260826145848_shrink_doc_need_ceiling_to_read_default.sql lines 99-118). The prerequisite ceiling fix IS in effect: only 94 of 1,684,087 rows in public.doc_need_matches sit at distance >= 0.30. Measured top-K impact today (prod, row_number() over (partition by document_node_id order by distance)): K=20 keeps 330,639 of 1,684,087 rows across 21,151 documents — an 80.4% row cut, larger than the issue's ~865K -> ~316K projection because the live table is 1.68M rows, not 865K. NAMING DEFECT in the issue: the store table is public.doc_need_matches (2.19 GB, second-largest relation); public.library_doc_needs is NOT a table — it does not exist as any relation in any schema — it is the reader RPC (function). Retitle before acting.

Open questions — what the audit could not settle
  • #3495 Whether the residual cost is the re-keyed typed-item upsert specifically, versus the two sibling costs filed as #3531/#3532. Close with: SELECT tenant_id, phase, avg(duration_ms) FROM (per-phase substep timings inside ingestion_runs.metadata) WHERE started_at > now() - interval '3 days' GROUP BY 1,2 ORDER BY 3 DESC — i.e. read the graph_materialization substep breakdown out of ingestion_runs rather than the whole-run duration used above.
  • #3531 No wall-clock figure for the digest itself — the issue's own [HYPOTHESIS]. Closes with: `EXPLAIN (ANALYZE, BUFFERS)` on the five-CTE query with tenant e9d96b09-4e76-4bd6-b136-e36515d79695 via Supabase execute_sql (read-only, but it does run the aggregates).
  • #3532 Which substep dominates on Regina — `phase_times` only carries the phase total; the propose/judge/write_edges split lives in the substep timing payload. Closes with: `select run_data->'substeps', phase_results->'graph_cross_conv_matching' from public.ingestion_runs where id='a8781a46-b93a-482a-a462-8476d7a5e6a0'`.
  • #3601 K=20 safety was asserted from 'worst rank any caller reads is 7' in the exploration doc; not re-derived here. Close with, in prod: with r as (select document_node_id, row_number() over (partition by document_node_id order by distance) rn, distance from public.doc_need_matches) select max(rn) from r where rn <= 20 and distance < 0.30 — plus a census of every caller of library_doc_needs (git grep -n 'library_doc_needs' origin/main -- client backend) to confirm no non-top-1 reader exists.
Not yet assessed — filed after the audit
  • #3933 [fold-efficiency] a completing Regina fold trips the 30-min materialize-lease PAGING…
  • #3987 [fold-efficiency] 24% of Regina's needs are re-embedded daily with no content change
  • #3988 [fold-efficiency] Four hand-written copies of the HNSW probe shape — collapse onto one…
  • #3993 [fold-efficiency] Remove the doc<->need K cliff with enable_sort=off — K=300+ keeps the…
  • #4186 [L1-g1-w6,L1-g6-w1,L1-g6-w4] [fold-efficiency] Regina's fold phases outlast the gap…
  • #4310 [fold-efficiency] aboutness densification still falls back to a full tenant scan on ~3…
  • #4312 [fold-efficiency] the provider demoter can still strip structured calls of every capable…
  • #4325 [fold-efficiency] a third of Cloudflare conversation_extraction calls on v4-flash-0731…
  • #4573 [fold-efficiency] OpenRouter roster drift: disclosed-redundancy — openai/gpt-5.6-luna
  • #4574 [fold-efficiency] OpenRouter roster drift: structured-capable — openai/gpt-5.6-luna
  • #4583 [fold-efficiency] ordered backlog — take the first unticked item
  • #4591 [fold-efficiency] OpenRouter roster drift: superseded — deepseek/deepseek-v4-flash-0731
  • #4609 flylogs.py needs --until/resume support to census OOM events over a wide window
  • #4647 [L1-g6-w1,L1-g6-w4] fold-efficiency: measure the worker out-of-memory kills (flylogs…
  • #4832 [fold-efficiency] OpenRouter roster drift: unused-provider — azure/swedencentral
  • #5088 [fold-efficiency] route GIL-bound CPU work off the default executor, with one enforcement…
  • #5090 Health shows when a fold failed because the database pool could not connect
  • #5397 [L1-g1-w6,L1-g6-w1,L1-g6-w4] [fold-efficiency] graph materialization restarts from zero…
  • #5567 [fold-efficiency] OpenRouter roster drift: unused-provider — NextBit
  • #6046 fix(backend): an expired source_enrichment lease waits for the slowest worker in the…

6 assessed issues closed since the audit and are not shown — a closed issue's evidence is history, and printing it as current is how a page starts lying.

Standing rulings

History

The founding exploration (2026-08-26)

docs/exploration/20260826-regina-fold-cost-attribution.md — scoped to "can we defer document extraction out of the fold" and answered that the question is moot: extraction is per-source (ADR-118's elastic pool), never in the fold, and the fold overruns the ceiling without it (the 08-25 run died at 4.33h with no document-extraction phase). Cost attribution: the two graph phases are 83.4%. The exploration also found the enqueue gap (#3600), measured the HNSW alternative (14×/probe, two silent traps), and ranked the precompute-sizing alternatives that became #3601/#3602 after PR #3594 landed option 1. Its §6 explicitly does NOT recommend: deferring document extraction, raising the 3h backstop, starting the extractor pool (already started), or scaling out (19/19).

The enqueue gap closed (2026-08-31)

#3600 was closed after a verification pass, not a triage pass — triage had proposed 26 closures on this batch of issues and verification held 13 of them, this one included, confirmed by an adversarial critic that re-derived the p0/p1 verdicts rather than accepting them. The fix: doc_extract_drain.py's _arm_catch_up_folds calls enqueue_cross_source per tenant per batch, on both normal and exception exits from run_drain. Verified reachable (scheduled by both worker.py and extractor.py) and verified exercised (30 work_queue rows with trigger=doc_extract_batch, newest 2026-08-31).