Mindbridge Β· track status

REGINA-DEDUP

The 772d15641 deploy (the #3128 conversation-scoped content-node re-keying) made graph_materialization's skip-when-unchanged stop firing: every run became a full-tenant scan, timed out after 1,508–2,516s, and 63 of 63 runs failed β€” every downstream phase blocked for Regina from 2026-08-16, ingestion paused, while the dedup pass had already merged nodes that should never have merged. This track is the gated repair: pause, capture undo, gate every merge surface on verified identity, re-materialize, re-merge, verify, unpause.

Incident onset 2026-08-16 Β· diagnosis + recovery plan committed 2026-08-20 (PR #3494, plan in docs/plans/20260820-regina-dedup-recovery/) Β· code phase complete 2026-08-21 (five PRs) Β· repair executed 08-22β†’08-25 Β· ingestion RESUMED 2026-08-25, owner-approved, healthy since (34 completed / 0 failed at first check) Β· page written 2026-08-25 from the fleet reports, the merged PRs and the repair sessions' own measurements. Counts resolve live at build.

Next β†’ Post-repair verification tails, not new repair. The re-embedding pass is the most urgent: ~66,000 stale embeddings were discovered across Regina's graph the night of 08-25 β€” the re-embed ops script is built and integration-tested but not yet applied. Alongside it: watch the ~8,350-document OCR backlog draining at ~3/min (it may be the gated ~23,000-doc extraction pass running without the sign-off it was waiting for β€” confirm intent), then the owner decisions queued below (#3497 retention, #3553's ~948 held-out merges, #3554's skipped-types ruling).
10 open β€” the incident itself is REPAIRED: the wrong merges are gone (redirect table at 0 rows for person/company/product/location after wave 1+2 delete-and-rematerialize; 10,106 legacy redirects deliberately left for wave 3), the re-merge ran under the identity gate (66 real merges on company/location/product), 4,029 items were recovered into Regina by the item-recovery backfill, the #3128 leak remediation applied (1,702 nodes nulled / 21,368 sibling edges / 13,138 duplicate bare nodes deleted fleet-wide), and Regina's ingestion was unpaused 2026-08-25 after a 3-day PR audit and an 8-session poll found no full re-materialization outstanding. What is open is the tail: verification (embeddings, OCR backlog), the fold-efficiency program #3495 spun out of the incident (and grew its own track page 08-26), two dedup-judge quality gaps, and one deep-dive ruling.
10open issues
63/63runs failing at onset (08-16)
0wrong-merge redirects after repair
13,138duplicate bare nodes deleted (08-25)
0 / 209carrierless among new enterprise LABELED nodes, 24h (08-26)
~66,000stale embeddings discovered (08-25, unrepaired)
Open work Why this order Rulings History

Open work

Verification tails β€” the repair is done, its residue is not

p1 no issue yet β€” re-embed the ~66,000 stale embeddings discovered post-resume script built + integration-tested, NOT applied Enterprise
Discovered the night of 2026-08-25 while watching the post-resume fleet: ~66,000 stale embeddings across Regina's graph (the repair's delete-and-rematerialize passes left embeddings behind β€” re-embedding was known unfinished as early as the 08-22 verification, which measured 9.5–47% NULL embeddings per type, worst company at 47%). An ops re-embed script was built and integration-tested in-session; its push/PR was still in flight at session end. Stale embeddings degrade every similarity surface β€” matching, dedup candidates, the viewer-truth detector β€” so this outranks the decisions below.
p1 no issue yet β€” the ~8,350-document OCR backlog may be the gated extraction pass running unsigned confirm intent β€” it drains at ~3/min Enterprise
Post-resume observation found a document-OCR backlog draining at ~3 documents/minute. The maildir attachment recovery (PR #3559, executed 08-25 β€” 76,228 attachments recovered) left ~23,000 documents to extract, each an OCR/LLM operation, explicitly awaiting an owner cost decision. The backlog's shape suggests that gated pass may be running anyway, triggered by the resume. Either confirm it is wanted or stop it; the decision was the point of gating it.
#3490 graph_materialization skip-when-unchanged not firing, full-tenant scan times out β€” blocks all downstream phases for Regina ingestion resumed & healthy β€” label pull owed on a measured fold Enterprise
The incident's founding issue, fixed in layers: the PG_MATERIALIZE_STATEMENT_TIMEOUT_MS route-around (PR #3494), the fold's document-read delta scoping (PR #3521 β€” see #3495 below), and the statement-timeout lift on the repair batches (PR #3547, verified live against the pinned worker). state:not-in-effect stays until a post-resume tenant-wide fold completes within timeout without the route-around crutch β€” the “34 completed / 0 failed” post-resume check is per-source ingestion health, not yet that proof.

The fold-efficiency program β€” #3495 and its waves

↗️ The cost class grew its own track on 2026-08-26: the fold-cost attribution exploration (docs/exploration/20260826-regina-fold-cost-attribution.md β€” document extraction is not in the fold; two graph phases are 83.4% of it) and the doc-needs sizing program (ceiling fix merged as PR #3594; the top-K store and HNSW build filed) now live on the FOLD-EFFICIENCY page β€” including the silent doc-extract enqueue gap found on the way (issue #3600, closed 2026-08-31 β€” detail in History). This cohort keeps the re-keying follow-up and the incident's own measured costs.

p1 #3495 Fleet-wide fold-cost step from 772d15641's per-conversation content-node re-keying delta-skip landed + verified live; waves A/B remain All
The system follow-up to #3490. The ruled direction was delta-skip, NOT budget-sizing, and the first lever shipped as PR #3521: an independent documents_synced_at watermark scopes the fold's document reads to the delta (round 1's panel caught a silent-data-loss cut β€” it borrowed a signal that could not see document-only changes β€” and the watermark is the fix). Verified live on the contrast tenant: documents substep 18.34s β†’ 5.96s (βˆ’67%), doc nodes per delta fold 456–462 vs 1,508–1,513 on fulls; the orchestrator verdict over all 14 post-deploy runs was SOUND, VERIFIED LIVE, KEEP. The survey that founded it measured the person+document substeps at 75.5% of a zero-change delta run. Wave A closed 2026-08-27 β€” issue #3502 (CI wiring) and issue #3501 (watermark flake) both shipped and verified live (detail in History). Remaining waves: B (the persons leg β€” design-first, owner-gated, because naive scoping forces every delta run to full; the typed-item leg), C (Regina-scale proof β€” largely overtaken by the repair itself). Two further costs it measured are filed as #3531 and #3532.
p2 #3497 graph_merge_snapshots retention/compaction decision options analyzed β€” owner decision owed Dev only
Consciously deferred at PR #3500 (the undo-capture table) and not silently dropped. A read-only options lane analyzed the choices with the recommendation Option 3: keep-all now + a re-measurement gate; a claims-verifier pass reproduced 29/29 of its claims. The decision is the owner's β€” the snapshots ARE the repair's undo trail, so premature compaction deletes the safety net while the wave-3 legacy backlog (10,106 redirects) is still unrepaired.

Judge quality β€” what the repair exposed

p1 #3553 identity gate: ~948 judge-approved merges held out for missing verified identifier (company hit hardest) owner ruling owed β€” real merges or noise? Enterprise
The identity gate (PR #3506 β€” verified-signal partition: emailβˆͺJID sets, the two-branch WORKS_AT rule, BASED_IN corroboration) refused ~948 merges the LLM judge had approved, because no verified identifier corroborated them. Two readings: the gate is doing its job against a label-happy judge, or real entities that share no key are being kept apart. Cross-listed on AMA-TRUST; the decision belongs to whoever owns judge precision.
p2 #3551 product-type dedup judge: label-only precision/recall gap on the Regina catalog candidate domain-hint fix needs measurement Enterprise
Filed off the live incident that also produced PR #3527's weight-token gate (a real drain had left 30 of 65 bad product merges with zero undo trail). The judge merges products on bare string similarity β€” the weight/pack-size gate fixes the weight-mismatched SKU class only, and unrelated products merging on string similarity is the separate, larger judge defect this issue carries. Cross-listed on AMA-TRUST.
p1 #3554 graph_dedup skips 11 content-node types entirely since the #3128 cutover β€” intentional or regression? deep-dive owed β€” needs a ruling, not a fix Enterprise
Since the conversation-scoped identity cutover, dedup processes none of the 11 typed content classes. Either that is intended (same text in two conversations is supposed to be two nodes now) or it quietly re-fragments the graph. The sibling-linker measured the stakes β€” 67% of 69,396 content nodes are same-text fragments, ~24,600 of which would collapse to one group each β€” but linking (SAME_CONTENT_AS) is the shipped answer to that, not merging. Cross-listed on WIRE-TRUTH, where the full row lives.
Why is it in this order?

Repair first, verification second, decisions third. The repair itself is done β€” so what orders the remainder is what still degrades answers silently. The stale embeddings outrank everything because every similarity surface reads them (the viewer-truth detector's “embedding-bounded floor” caveat and the dedup judge itself both inherit the staleness). The OCR-backlog question outranks the decisions because ~23,000 documents of spend may be running without its sign-off. The fold-efficiency program keeps its #3495 priority because it is what stops the incident class from recurring on the next deploy. The judge-quality cluster sorts last because it is real but was ALREADY the pre-incident state β€” the repair did not create it, it exposed it.

Deliberately ignored: priority labels alone; filing order; the sunk cost of the repair itself.

What the audit found β€” generated

⚠︎ The prose above cites 1 issue that is no longer open

#3490. A closed issue named as current work is the drift this section exists to surface β€” re-derive the prose, or move the reference into History.

Evidence below is generated from the read-only audit of 2026-08-29 and reconciled against the open issues at build time. Every claim cites the issue it came from. Nothing here is a decision β€” rulings stay in the prose above, which is written by hand.

#3007 [L2-g4-w6] Name sanitization carries three CLAUDE.md-banned lexical heuristics (hardcoded…codehigh confidence

All three constants are present verbatim on origin/main backend/app/core/constants.py: line 192 `NAME_MAX_ARABIC_WORDS = 4`, line 201 `NAME_GROUP_TAG_BOILERPLATE = frozenset({"whatsapp","chat","group","the","and","for","with","of","in","to"})`, line 207 `NAME_COMMON_FIRST_NAMES` (Latin-script transliterations only). All three are consumed on the live ingestion path: `NAME_MAX_ARABIC_WORDS` at backend/app/orchestration/phases/name_resolution_phase.py:154 inside `_is_arabic_phrase`, wired into the phase's predicate list at line 203; the other two at backend/app/services/name_resolution/resolver.py:161 and :163 inside `NameResolverService.extract_group_tags`, itself called from name_resolution_phase.py:698 and resolver.py:746. Consumer census: `git grep -n "NAME_GROUP_TAG_BOILERPLATE|NAME_COMMON_FIRST_NAMES|NAME_MAX_ARABIC_WORDS|_is_arabic_phrase" origin/main -- backend` β€” the hits above plus backend/tests/unit/services/test_name_sanitization.py, which PINS the current behavior (line 105 asserts 'Ali Investment Group' keeps 'ali' via the first-name list). No replacement statistical/graph/LLM mechanism exists anywhere in that path.

#3495 [fold-efficiency] the fold's dominant cost is graph_cross_conv_matching +…commithigh confidence

Only the DOCUMENT leg was scoped to delta: commit 900ef0436 on origin/main ([fold-efficiency] fix(backend,db): scope fold document reads to per-conversation delta with an independent sync watermark (#3495) (#3521)). Its own commit message states 'PersonMaterializer is intentionally left untouched ... Flagged as owner-gated follow-up, not solved here.' No commit on origin/main since 2026-08-18 addresses the typed-item leg (git log origin/main --since=2026-08-18 -i --grep 3495 returns exactly two commits: 900ef0436 and dc71c85b1, the latter being CI wiring for #3502). Prod corroboration (public.ingestion_runs, last 5 days, grouped by tenant/day): the cost is still fleet-wide and heavy β€” 2026-08-28 tenant e9d96b09 (Regina) 637 runs / 51 failed / avg 1,724s / max 33,117s; tenant be0f3db0 178 runs / 18 failed / avg 559s / max 25,944s; 2026-08-27 avg 2,620s Regina and 2,280s on tenant 23d86e0d. Caveat: ingestion_runs durations are whole-run, not the graph_materialization substep alone.

#3497 graph_merge_snapshots retention/compaction decision (deferred at PR-2, regina-dedup)prodhigh confidence

No retention exists and the table is growing fast. (1) SELECT table_name FROM public.retention_policy_registry β†’ 12 rows, graph_merge_snapshots is NOT among them (ama_turns, chat_threads, checkpoints, connector_staged_events, doc_need_matches, help_conversations, help_escalations, help_feedback, ingestion_runs, llm_usage_events, posthoc_resolution_proposals, work_queue). (2) The prod cron census (33 rows of cron.job, all enumerated) contains 12 prune-* jobs, none for graph_merge_snapshots. (3) Growth, measured: SELECT count(*), count(DISTINCT tenant_id), min/max(captured_at), pg_total_relation_size β†’ 5,231 rows / 5 tenants / 2026-08-21 07:09Z β†’ 2026-08-29 11:18Z / 126 MB. That is 228 rows and 6.0 MiB at the issue's 08-22 analysis comment β†’ 5,231 rows and 126 MB seven days later: ~23x rows, ~21x bytes, on metered gp3 disk that does not shrink on its own.

#3531 The ADR-111 skip-check digest is itself O(tenant) β€” a 5-way full-tenant re-aggregate runs…codehigh confidence

origin/main:backend/app/slices/graph_building/repositories/node_index_repository.py:106-145 β€” `_PERSON_ATTRIBUTE_DIGEST_SQL` still runs five independent full-tenant aggregates (person_valuable_message_counts, person_impersonation_verdicts, person_automated_verdicts, whatsapp_jid_mappings, person_identities), each `string_agg`+`md5` with no watermark/delta predicate. Reachable on every fold: `graph_materializer.py:409` and `materialize_scan_planner_service.py:141` both call `person_attribute_input_digest()` unconditionally. `git log origin/main -- node_index_repository.py` shows the newest touch is #3355 (2026-07, WIRE-TRUTH), nothing after the 2026-08-22 filing.

#3532 graph_cross_conv_matching costs more than the entire fold on healthy tenants (propose…prodhigh confidence

Query on prod `public.ingestion_runs` (phase_times JSON), runs since 2026-08-22: cross-conv vs fold seconds β€” Regina e9d96b09: 5403 vs 726 (08-29 08:06Z), 4622 vs 1445 (08-29 04:09Z), 5694 vs 372 (08-28 23:11Z), 5315 vs 2038 (08-28 17:39Z); contrast tenant 23d86e0d: 128 vs 41, 109 vs 40, 97 vs 30. The premise holds and has WORSENED: on Regina cross-conv is 3-15x the fold and is measured in thousands of seconds. Code side: `git log origin/main --date=short -- backend/app/slices/graph_building/phases/graph_cross_conv_matching_phase.py` shows only #3594 (doc_need ceiling) and #3526 since filing β€” no diagnosis or perf change to propose/judge/write_edges.

#3533 Bot-sender lifecycle after the repair: classifier bidi false-positive rule, ingestion…codehigh confidence

Item 1 (bidi false positive) NOT fixed: origin/main:backend/scripts/regina_dedup_classify_bot_senders.py:84 still `_INVISIBLE_CATEGORIES = frozenset({"Cf"})`, which includes RLM/LRM/ALM, and line 137 fires `matched_rule="invisible_unicode"` on that alone with no corroboration requirement. Newest commit on that file is c7f9e4b18 (2026-08-21, PR #3513) β€” predates the 2026-08-22 sign-off that filed this. Item 2 (skip-list) absent: `git grep -rn "bot_" origin/main -- backend/app` (population = every file under backend/app on origin/main) returns only Slack connector bot_token/bot_id hits, no sender skip-list; `git grep -rn suppress -- graph_building/aggregators/person*.py` returns nothing. Item 3 (monitor) absent: no bot-mint check in `backend/app/orchestration/invariants/checks/` (10 files enumerated, none bot-related) and none of the 33 pg_cron jobs or 6 GH Actions crons drives one.

#3551 [L2-g1-w7] product-type dedup judge: label-only precision/recall gap on Regina catalog,…codehigh confidence

The production judge system prompt is unchanged since the issue was filed: `backend/app/slices/graph_building/services/dedup_judge.py` `_SYSTEM` (module read on origin/main) contains no domain-hint sentence β€” no "shared brand/base-name" clause, no size/packaging-qualifier clause; the same/sub_type/different definitions are the pre-existing generic ones. `git log origin/main --oneline --since=2026-08-23 -- backend/app/slices/graph_building/services/dedup_judge.py` returns no commit touching that file (the four commits in that window touch concurrency knobs, identity-signal chunking, doc-need ceiling and doc-project matching). The measured +0.08 precision / +0.16 recall arm has not been adopted, and none of the four stated next steps (larger stratified sample, held-out eval, panel, Finding-1 inconsistency probe) has landed.

#3553 [L2-g1-w7,L2-g6-w1] identity gate: ~948 judge-approved company/product/location merges…codehigh confidence

The gate is unchanged and the behavior the issue describes is the coded design, not a regression: `backend/app/slices/graph_building/merge_partition_gate.py` module docstring Β§4.1 step 2 β€” "If any member carries identifiers, members with NO resolvable identifier at all do NOT ride in for free β€” each becomes its own singleton holdout" β€” implemented at lines 182-198 (`_disqualified_holdouts`, `singleton_holdout_ids`). Identifier sets are per-type (docstring line 7: person emailβˆͺJID; company employee email domains; location address tokens) with no company-specific alternative signal β€” no corroboration floor or structural check appears in the module (grep over the file for company/node_type/holdout/singleton returns only the docstring mentions and the generic holdout machinery). `git log origin/main --since=2026-08-23 -- backend/app/slices/graph_building/merge_partition_gate.py` shows no commit. The issue itself asks for the owner's judgment on whether company gets a less email-dependent verification signal; that ruling has not been made.

#3554 graph_dedup skips 11 content-node types entirely (opportunity/need/problem/solution/...)…codehigh confidence

origin/main:backend/app/slices/graph_building/constants.py:364-374 β€” V2_NODE_DEDUP_THRESHOLD still holds exactly 8 keys (company, person, topic, skill, location, product, interest, trait); none of the 11 content types. origin/main:backend/app/slices/graph_building/phases/graph_dedup_phase.py:562 still reads `dedup_types = [t for t in present_types if t in V2_NODE_DEDUP_THRESHOLD]`, so those types are silently filtered out of every dedup pass. Prod re-measurement (supabase execute_sql, schema-qualified public.nodes_v2, 2026-08-29): the 11 types now total 46,276 nodes; grouping by canonical_name with the trailing `::<uuid>` conversation discriminator stripped gives 5,137 nodes sitting in a group with a sibling (11%). The leak half: bare (non-suffixed) nodes = 3,185, of which only 17 carry >1 provenance_conversations; only 2 nodes fleet-wide carry the `properties.leak_fix_batch` stamp (both with source_message_id NULL), so the null-the-anchor remediation described in the comments was never run at scale β€” the population it targeted has instead shrunk on its own (2,373 bare multi-conversation nodes at filing vs 17 now).

Open questions β€” what the audit could not settle
  • #3007 The actual blast radius β€” how many person names in production had a token stripped as a group tag that was really a first name. Close with (Supabase MCP execute_sql): `select p.tenant_id, count(*) from person_identities p join sources s on s.tenant_id = p.tenant_id where s.source_type='whatsapp' and p.canonical_name ~ '[Ψ€-ΫΏ]' group by 1;` as a denominator, then compare `person_identities.canonical_name` against the raw sender strings in `messages.sender` for the same tenant to count divergences.
  • #3495 Whether the residual cost is the re-keyed typed-item upsert specifically, versus the two sibling costs filed as #3531/#3532. Close with: SELECT tenant_id, phase, avg(duration_ms) FROM (per-phase substep timings inside ingestion_runs.metadata) WHERE started_at > now() - interval '3 days' GROUP BY 1,2 ORDER BY 3 DESC β€” i.e. read the graph_materialization substep breakdown out of ingestion_runs rather than the whole-run duration used above.
  • #3497 Whether the 08-21β†’08-29 growth is steady-state or another repair burst (the 08-22 analysis showed 224 of 228 rows came from a single 1h41m re-merge window). Close with: SELECT date_trunc('day', captured_at) d, tenant_id, count(*), count(DISTINCT merge_batch_id) FROM public.graph_merge_snapshots GROUP BY 1,2 ORDER BY 1 DESC;
  • #3531 No wall-clock figure for the digest itself β€” the issue's own [HYPOTHESIS]. Closes with: `EXPLAIN (ANALYZE, BUFFERS)` on the five-CTE query with tenant e9d96b09-4e76-4bd6-b136-e36515d79695 via Supabase execute_sql (read-only, but it does run the aggregates).
  • #3532 Which substep dominates on Regina β€” `phase_times` only carries the phase total; the propose/judge/write_edges split lives in the substep timing payload. Closes with: `select run_data->'substeps', phase_results->'graph_cross_conv_matching' from public.ingestion_runs where id='a8781a46-b93a-482a-a462-8476d7a5e6a0'`.
  • #3533 Whether any bot person has actually been minted since the repair (item 2's urgency). Closes with: `select count(*) from nodes_v2 where tenant_id='e9d96b09-4e76-4bd6-b136-e36515d79695' and node_type='person' and created_at > '2026-08-22' and canonical_name ~* '(noreply|docusign|amazon|notification)'`.
  • #3551 Whether the three-contradictory-verdicts inconsistency still reproduces on live inputs. Close with: `select node_type, decision, count(*) from public.nodes_v2_dedup_verdicts where node_type='product' group by 1,2` plus a re-run of the judge on the three recorded Regina-400g pairs.
  • #3553 Whether the 948 holdout count still stands after the manual out-of-band drain. Close with the issue's own no-LLM query: cross-join cached `same` verdicts in `public.nodes_v2_dedup_verdicts` against `public.graph_merge_redirects` per node_type.
  • #3554 WHAT retired the ~2,350 bare multi-conversation nodes between 2026-08-24 and now is not established β€” no leak_fix stamp accounts for it (2 stamped nodes only), so a re-materialize/prune pass is the candidate but unproven. Close with: `select date_trunc('day', updated_at), count(*) from public.graph_merge_snapshots where created_at > '2026-08-24' group by 1 order by 1;` plus `git log --since=2026-08-24 --oneline origin/main -- backend/scripts/ops/leak_fix backend/app/slices/graph_building/repositories/node_prune_repository.py`. Also unclosed: whether the 17 residual bare multi-conversation nodes are still served (needs the audience_gate trace per node).
Not yet assessed β€” filed after the audit
  • #6080 Relationship-name prefetch (#6050): no chooser-call burst on Regina full fold 511011b7…

3 assessed issues closed since the audit and are not shown β€” a closed issue's evidence is history, and printing it as current is how a page starts lying.

Standing rulings

History / forensics

The repair, as it actually ran (08-20 β†’ 08-25)
  • Code phase (08-20β†’21, five PRs): the diagnosis and route-around (PR #3494) Β· pause-gap guards + the identity-signal index + seven audit scripts (PR #3503) Β· merge-undo capture, graph_merge_snapshots (PR #3500) Β· the identity gate at all four merge surfaces, type-enforced so skipping it is a type error (PR #3506) Β· the out-of-band repair runner + canary battery (PR #3513). Panel totals across the program: 24 lens-runs, 20 majors found and fixed, ~70 mutation kills.
  • Hardening off live incidents (08-21β†’24): the product weight-token gate + a real merge-undo restore tool, after a drain left 30 of 65 bad merges with no undo trail (PR #3527) Β· unioning provenance_person_ids onto survivors, ending a permanent identity drop on every merge ever run (PR #3530) Β· the statement-timeout lift, verified live against a real 366-key person-type delete timeout (PR #3547) Β· the content-sibling linker and its ragged-array hotfix (PR #3567, PR #3570) Β· the mixed-splittability edge splitter (PR #3571) Β· the single-conversation duplicate collapser (PR #3573).
  • The repair itself: rehearsal green 08-22 (battery 86/0 in 62s; the full pipeline green end-to-end twice on the dev stack; a full-scale tenant-copy rehearsal was DESCOPED on a measured PK-collision blocker). Wave 1+2 delete-and-rematerialize ran against prod: redirects 0 rows for all four types, Kareem's case spot-verified correct, 10,106 legacy redirects left for wave 3. The full re-run (08-24): fold ~2.5h, one aborted attempt on a legitimately pre-merged key retried with --suppress-keys, 66 real merges confirmed complete on company/location/product; person's zero-merge result was never proven safe in-run (its follow-up check hung) and person merges remain the cautious class. The item-recovery backfill (08-24) recovered 4,029 items into Regina (3,278 typed, 751 relationships; 563 person↔person edges) with the accounted figure matching prediction.
  • The #3128 remediation (08-25): the leak-nulling run completed β€” 1,685 Regina nodes (1,702 fleet-wide) nulled fail-closed, 21,039 sibling edges written on Regina (21,368 fleet-wide), 13,132 Regina duplicate bare nodes deleted (13,138 fleet-wide), with a 4,187-row doc_need_matches-blocked follow-up recorded. Then the resume: ingestion_paused true β†’ false, owner-approved, verified in the same query; post-resume reads 34 completed / 0 failed; the carrierless count held at 3,574 ≀ the 3,588 baseline that day, and #3488's close on 08-26 later proved the writer outright clean (209 new enterprise LABELED nodes in 24h, 0 carrierless) with the standing population reduced to 112 backfill-impossible rows by this campaign's own duplicate deletes.
  • What the plan intended that did NOT happen: the formal Gate-2 assert package and the seven audit censuses were never presented; the owner's resume decision rests on the audit + poll instead. The full-scale rehearsal was descoped (synthetic-only). Two follow-ups the fleet named as owed were filed (#3501, #3502) and the audit's company-arm join-case divergence was noted but never filed.
Fold-efficiency wave A β€” closed 2026-08-27
  • #3502 tests/invariants/ collected by no CI job β€” CLOSED. Shipped as PR #3592 (squash dc71c85b12, merged 08-26): the 43-item invariants suite now runs as a third serial pytest pass inside integration-tests.yml's tests step, so a failure reddens the required Backend: Integration Tests status; a DB-free wiring-pin test structurally guards against the step being moved or deleted. Verified live in the real CI runner: 40 passed / 3 skipped, matching the pre-change local baseline.
  • #3501 watermark-boundary flake β€” CLOSED. Shipped as PR #3593 (squash 22311e121, merged 08-27): the two delta-skip tests now seed the watermark explicitly to max(messages.updated_at) + 1Β΅s after pass 1, removing the dependency on sibling-backend quiescence that the ADR-112 C2 DB-wide min(xact_start) horizon created. Verified deterministic 8/8 post-fix (the panel reproduced the pre-fix flake live by holding a transaction open).
The incident, measured (from the #3494 diagnosis)
  • Pre-deploy delta runs: 335–464s (median 350s). Post-deploy: zero completions β€” all 63 runs fail QueryCanceledError after 1,508–2,516s, burning statement-timeout budget on every attempt. Full folds before the incident: 1,356–1,941s.
  • The affected plan at its peak: 6,210 keys (person 383, company 3,130, product 1,296, location 1,401); person redirects 226β†’228 between passes; census before delete: 3,632 nodes / 147,660 edges; graph_merge_redirects 14,299 (4,193 from the incident's own batches + 10,106 legacy); backups taken of all 14,299 + 56,394 rows, byte-exact.
  • Classifier CSV: 228 rows β€” 36 bot, 420 unknown at classification time; 35 bot sides and 30 person keys suppressed by the signed Gate-1 decision; 35 nodes carried NULL embeddings pre-repair.