Every file a message carried, present and findable β a message can carry N files but the substrate keeps one, and the four defects run from upload through audience to serving and recovery.
π’ The recovery ran to completion on 2026-08-25 (finished cleanly 06:54 UTC), using the script shipped as PR #3559 (merged 2026-08-24, executed as an ops action). Production outcome, measured after the run: 76,228 attachments recovered β messages with attachments went 954 β 22,498, attachment rows 3,451 β 79,679 (this pass wrote 42,655), 35,360 distinct content digests. The fail-closed message_hash gate refused 1,221 hash-mismatched candidates β the safety property held on real data. One attempt was killed mid-run by a deploy; the watermark made the re-run safe, which is the idempotency claim proven live.
π’ The retention-window pause was reverted β connector_staged_events retention is back to 30 days, verified safe against the held backlog: the prune only deletes processed_at IS NOT NULL rows, and the 111,952 held-back unprocessed emails were untouched.
π What remains, and it is the whole remaining row: none of the recovered attachments is findable yet β DOCUMENT nodes are unchanged at 12,188, and surfacing them needs ~23,000 document extractions, each an OCR/LLM operation, a cost decision explicitly awaiting the owner. The serving cap (#3326) and the audience fork (#3367) both sit in front of that pass. Attachments lost to the %-filename defect (PR #3537) are not recovered by this run β they left no rows, so the watermark excludes them. The issue stays open with state:not-in-effect until the recovered documents are extracted and findable; its named acceptance case (Regina's invoice retrievable by its content) is still unmet. Co-tracked with AMA-TRUST.
π’ Fixed in PR #3537 (merged 2026-08-23) β a one-character fix: re.compile(r"%") in build_storage_key replaced % with _, and the storage layer treats a literal % as a format specifier, so every attachment whose filename contained one failed its upload silently and permanently. Found during the recovery's dry-runs. The panel refuted a more aggressive allowlist version that would have re-keyed 988 Arabic-named attachments β measured on prod 2026-08-19: 4,517 stored keys, 0 containing %, 1,788 with spaces.
π Attachments already lost to this defect are not recovered by merging it β they have no rows, and for messages that gained rows for their other attachments the #3327 watermark now excludes them. That repair is tracked separately and has no issue yet.
Two locally-raised extraction failures permanently retire the attachment they touch, neither a provider problem. Measured on prod 2026-08-30.
1. xlsx text-entity extraction truncates at max_tokens β the dominant, growing loss. Regina: 10,135 xlsx objects, 749 permanently retired (7.4%), +59 in the last 24h. Log signature: text-entity LLM failed β¦ structured output truncated at max_tokens (a second variant surfaces as a pydantic ValidationError). Start at xlsx_extractor.py and the max_tokens values in llm_provider.py (several providers pin 30000, one pins 16384) β the log line names three different model strings, so the routing is worth reading before changing a number. Whether the fix is a larger budget, chunking the workbook, or both is open.
2. Image decode ceiling. A second, smaller-volume cause with the same permanent-retirement shape.
The reference-hygiene cohort (stale attachment rows, stranded objects, the writer SET ROLE bypass) closed 2026-08-26 β 09-01 in full β see History.
The defects are a chain, not a menu β each rung breaks ties left by the one above it. The first two rungs were climbed on 2026-08-17, which is what moved the head of this page.
What this order deliberately does NOT use:
#3675. A closed issue named as current work is the drift this section exists to surface β re-derive the prose, or move the reference into History.
Evidence below is generated from the read-only audit of 2026-08-29 and reconciled against the open issues at build time. Every claim cites the issue it came from. Nothing here is a decision β rulings stay in the prose above, which is written by hand.
Prod (Supabase execute_sql, 2026-08-29): message_attachments = 88,278 rows over 25,950 distinct messages β the 2026-08-25 recovery pass is confirmed landed (issue comment claimed 79,679). Surfacing has since ADVANCED past the comment: nodes_v2 where node_type='document' = 29,849 (comment said 12,188; note the enum is lowercase 'document', a query on 'DOCUMENT' returns 0 and reads as "nothing surfaced"). document_extraction_attempts = 90,394. Acceptance still incomplete: of 1,261 attachment rows named '%invoice%.pdf', only 445 have a nodes_v2 document node keyed by properties->>'file_path' = storage_key (35%). The named acceptance case is SPLIT: message b0b7dc11-a724-415f-bada-4a2e2576a315 carries 'August 2026 Regina for pasta food Industries_Sum.xlsx' at attachment_index 0 and '..._Invoice.pdf' at index 1, each with exactly one document node β structural items 1 and 2 met; the sibling message b9f073b9-882c-4720-b8a0-e827c2930f36 (March 2026, same pair) has BOTH attachments at zero document nodes.
2 assessed issues closed since the audit and are not shown β a closed issue's evidence is history, and printing it as current is how a page starts lying.
message_attachments shipped as PR #3378, and the historical loss is recovered by the maildir backfill (#3327) rather than by a migration β deliberately, because the recovery needs disk gating and a unit-cost decision a migration cannot carry (2026-08-17).The %-filename upload failure (PR #3537, merged 2026-08-23). Found in-flight during recovery dry-runs: every attachment whose filename contained % failed its upload permanently and silently. One-character fix; the panel refuted the aggressive allowlist variant that would have re-keyed 988 Arabic-named attachments.
The recovery script (PR #3559, merged 2026-08-24) and its execution (2026-08-25). The script pairs messages to retained raw .eml objects via Message-ID selects under a fail-closed message_hash gate (35/35 on the control sample); sizing measured on prod 2026-08-24: 260,922 raw objects (197 GB), 95,844 orphaned. It then ran as an ops action to completion: 76,228 attachments recovered, 1,221 hash-mismatches refused, one deploy-killed attempt re-run clean off the watermark. The retention-window pause that protected the population was reverted to 30 days afterwards, verified safe against the held backlog. What the run did NOT buy is findability β DOCUMENT nodes are unchanged, and the ~23,000-document extraction pass is an owner cost decision that now heads the queue above.
#3324 The substrate model (PR #3378, ADR-145 D1/D2). message_attachments keyed (message_id, attachment_index), NOT NULL FK ON DELETE CASCADE, UNIQUE (message_id, storage_key) so a re-run is refused by the database rather than by a guard. All three connectors (email, Slack, WhatsApp) emit N instead of reducing to one scalar β measured on Regina maildir, 57% of document-bearing messages had lost at least one attachment before this. The media scalars survive as a materializer-derived preview, which is what keeps "rows present, preview NULL" out of the schema instead of out of a code path. Covered by a real-DB integration test (three attachments β three rows in provider order, preview derivation, idempotent re-flush, duplicate refused) plus connector unit tests and a discriminating media_fanout test. Forward-only: it stopped the ongoing loss; recovering already-ingested messages is the maildir backfill (#3327) above.
#3326 The library one-document-per-message cap (PR #3398, ADR-145 D5). library_files_mat carried a single-column PRIMARY KEY (message_id), so a message with recovered attachments 2..N would fail its own refresh on a duplicate key β recovered documents would have landed in the graph and stayed invisible. Re-keyed on (message_id, document_node_id), reaching the ten live objects that resolved through the old key (library_files, search_library_files, similar_media, and seven more).
#3325 The Archive audience rule (PR #3366, ADR-145 D4). Most-restrictive-wins across every message sharing a document's content_hash, computed in the writer and intersected in the reader, unknown audience failing closed. Its round-2 lens is the reason this page gained rows rather than losing one: it found the identical single-anchor predicate in two other lanes and filed both separately β #3367 (the library_files serving surface, on this track) and #3368 (the Python AMA/Ask lane, on AMA-DISCLOSE) β rather than letting a one-surface fix read as track-wide. Both closed the next day, below.
π What the day actually bought, stated plainly. New mail keeps all its files, the library stopped capping documents at one per message, and the Documents Archive resolved audience correctly. No historical message had yet gained an attachment (that was #3327, 2026-08-25), and two serving lanes still disagreed with the Archive about who may read a document for one more day.
#3367 and #3368 closed together via PR #3408. Both had the same shape: library_files_mat.scope (#3367) and the Python Ask lane (#3368) each still resolved an unscoped document's audience from the single arbitrary-uuid source_message_id anchor the Archive fix had just stopped using β seven live RPCs gated on the library_files copy alone, so a document correctly withheld in the Documents Archive was still disclosed through Files, Needs and Memories. PR #3408 built one shared carrier-scope resolution behind every document-serving surface instead of patching each lane separately, which is what the CHOKE-POINT DIRECTIVE prefers. The two-surfaces-disagree window that opened when the Archive fix landed on 2026-08-17 lasted exactly one day.
#3463 The writer SET ROLE bypass, closed 2026-08-26 via PR #3591. A bare RESET ROLE in the message_attachments_writer migration left the role bookkeeping wrong, so the CLI-side invariant auditing the identity choke-point read false. The migration now ends in the role it found.
#3413 The stale-attachment-row reconciler, closed 2026-09-01. store_attachments upserted only the rows it was handed and never deleted a row that dropped out of a later flush, so a shrinking re-flush could leave a stale row sharing an attachment_index with a live one β pinned on the real DB, indices 0, 0, 1. New flushes now reconcile: a producer that saw a message's whole file set marks it authoritative, and the writer deletes rows that flush did not name, enqueuing their storage keys for the purge sweep in the same statement. The missing ADR-145 D1 reverse-invariant check (attachment_index_uniqueness) shipped registered. Still owed and tracked separately: a backfill over rows already stale on production, and reconciling the WhatsApp/ingestion-read-reconstruction lanes, which default to not-authoritative today.
#3430 The stranded-object purge enqueue, closed 2026-09-01 β same migration as #3413: superseded content-addressed objects are now enqueued for the existing hourly purge sweep (storage_deletion_queue + StoragePurgeService) at the delete site, atomically with the row delete.
#3668 closed the same day it was filed, via PR #3673. A billing 400, a 429, a 529 and a corrupt JPEG were all wrapped into MediaExtractorError at one seam and the dispatch error arm watermarked every cause as permanent, so a transient provider failure was never retried once the provider recovered β 6,830 objects retired all-time (Regina 89%), a floor rather than a count since every tenant shared one backfill-stamp first_seen. The fix classifies the failure before deciding whether to retire the object. Recovery of the already-watermarked population is separate and owner-gated (the revoke is lazy β it clears the watermark but schedules nothing).
π The fix's own systemic-failure detector deliberately excludes a failure the provider never answered (failure_signature() returns None for one, because retrying it fails identically) β which is exactly the shape of the two locally-raised causes filed the same day as #3675 and still open above.
The track is the attachment-model cluster (ADR-145), filed 2026-08-15 and pinned at rank 2: four defects spanning message storage, audience scope, the serving cap, and recovery of what was already dropped.
Every other stage was gated by the identity choke-point (#3323, ADR-145 D3, gate G0), which closed 2026-08-15. Six independent expressions answered one question β 'is this document already here?' β across four keys (content_hash, filename, storage_key, message row), coherent only because there is one attachment per message. The fix collapsed them onto one content_hash-keyed resolver, deleted the filename/storage_key/message-row sites rather than repointing them (repointing preserves the two-different-invoice.pdf-are-one-document filename collision), collapsed version predicates onto is_current_extractor_version, and shipped a check.py AST hook in the same PR.
Its constraint (panel finding F2) still binds the recovery: content_hash needs the bytes, so without a cheap byte-free negative check in front of the resolver an already-extracted attachment must be downloaded to decide to skip it.