Mindbridge ยท track status

ATTACHMENT-TRUTH

Every file a message carried, present and findable โ€” a message can carry N files but the substrate keeps one, and the four defects run from upload through audience to serving and recovery.

Re-measured 2026-08-16 from live gh issue list; counts resolve at build time. Verify before starting: the blocked label on #3324 was removed 2026-08-16 once its only gate (the identity choke-point, #3323) closed 2026-08-15 โ€” the only item on this track still carrying blocked is the recovery (#3327). This page has never been published โ€” publish fresh per docs/status/artifacts.json.

Next โ†’ #3324 message_attachments: a message can carry N files (ADR-145 D1/D2) โ€” the root cause that stops the ongoing silent loss. The index's NEXT column shows the audience rule (#3325) instead, because that issue is co-labelled track:ama-disclose, which sorts first in the queue's alphabetical track tiebreak โ€” this page's criterion wins here. label cleared 2026-08-16
5 open ADR-145 defects in one chain โ€” the model (upload + serving), the audience rule, and the recovery of โ‰ฅ71,440 dropped documents. For maildir nothing is lost โ€” every .eml is retained.
5open defects
57%doc-bearing maildir messages lost โ‰ฅ1
โ‰ฅ71,440dropped documents recoverable
273,648retained .eml objects (190 GB)
Open workWhy this orderStanding rulingsHistory

Open work

The model โ€” one message keeps N attachments
p1 #3324 message_attachments: a message can carry N files (ADR-145 D1/D2) stops the ongoing silent loss All

Message.attachment_storage_key is a scalar, so a message that carried five files keeps one. Three connectors reduce N to that scalar independently โ€” email_attachments._upload_first_attachment (Gmail/Exchange/maildir), slack/adapter._upload_first_attachment, and the WhatsApp path in batch_processor โ€” so no connector patch can fix it. Measured on Regina maildir: 57% of messages carrying a document lost at least one.

Scope: message_attachments keyed (message_id, attachment_index) โ€” storage_key, filename, content_type, size_bytes, content_id, is_inline_decoration; references and ordering only, never content (that constraint is what stops it becoming a second document_sheets, #3255). NOT NULL FK to messages ON DELETE CASCADE; UNIQUE (message_id, storage_key) so a re-run is refused by the DB, not a guard. Single-writer hook ships in the same PR; all three connectors emit N.

media_filename/media_type/attachment_storage_key demote to a derived preview = first document attachment, and a message with attachment rows plus a NULL preview must be impossible (media_type IS NOT NULL gates the whole library row). aggregate_documents adds attachment_index as a tie-break strictly after message_id โ€” before it would silently re-label existing nodes on the next fold. The media_fanout invariant (llm_noop_spend.py:78-105) must be fixed too: it divides LLM calls by a per-message count, so it would emit false churn alerts on exactly this multi-attachment population. Carries branch feat/attachment-multi.

๐Ÿ›‘ The blocked label this issue once carried was removed 2026-08-16. It named one blocker โ€” the identity choke-point (#3323, gate G0) โ€” which closed 2026-08-15. The issue is unblocked and now reaches the what's-next queue.

p1 #3326 The library caps documents at one per message via a PRIMARY KEY (ADR-145 D5) recovered docs would be invisible Latent

Ships with the recovery backfill, not after it. The serving layer caps documents at one per message โ€” and the cap is a PRIMARY KEY, not a query artifact: 20260624084354_library_files_mat_precompute.sql:33 does ALTER TABLE public.library_files_mat ADD PRIMARY KEY (message_id), never widened since. So removing the LIMIT 1 in library_files_source is NOT sufficient โ€” the refresh would fail on a duplicate key, and the recovered documents would land in the graph invisible to users, reading as 'the backfill did nothing' (the spreadsheet collapse, #3255).

Ten live objects resolve through that key: library_files ยท refresh_library_files_mat ยท search_library_files ยท similar_media ยท library_memories ยท library_memory_docs ยท library_person_files ยท library_place_files ยท library_doc_needs ยท open_items.

Also in scope: library_files_source's JOIN LATERAL โ€ฆ LIMIT 1 and its WHERE m.media_type IS NOT NULL gate โ€” an INNER join, so a message whose document node has not materialized drops from the view entirely; refresh_library_documents_mat stores attachment_storage_key as a single per-message scalar; message_file_type(media_type, media_filename, attachment_storage_key) is IMMUTABLE and types every attachment by the preview's extension. Render "+N more" on the conversation and library surfaces.

The audience rule โ€” most-restrictive-wins
p1 #3325 A document's audience can be decided by an arbitrary carrier message (ADR-145 D4) arbitrary uuid pick decides audience Enterprise

Gate G4 โ€” blocks the recovery backfill. The library_documents reader gate (20260801204638_library_documents_denormalized_scope.sql:244-253) resolves disclose_scope_is_visible(COALESCE(document_scope, carrier_scope), job_function, projects, tier). carrier_scope is cm.scope where cm.id = d.source_message_id, and source_message_id is min(doc_msgs[content_hash], key=str) โ€” an arbitrary-but-stable uuid pick among every message that ever shared those bytes.

document_scope wins when present โ€” the carrier is a COALESCE fallback, so this bites only documents carrying no document-level scope. For those, because disclose_scope_is_visible refuses a NULL scope for every viewer, the arbitrary anchor decides between visible-to-that-message's-audience and invisible to everyone. sensitivity_tier is document-level and is NOT affected โ€” only the scope/audience axis. Co-tracked with AMA-DISCLOSE.

Why in scope: pre-existing, but the recovery multiplies cross-message content_hash dedup โ€” attachments 2..N of busy messages are disproportionately files the graph has already seen โ€” so this work grows the affected population and cannot ship undecided.

Decision (ADR-145 D4): a document's effective scope becomes a deterministic function of all its sharing messages, defaulting to most-restrictive-wins โ€” the failure direction of guessing is over-disclosure. Verified compatible with the migration's deliberate 'truly unscoped stays open' arm.

Recovery โ€” the dropped attachments
p2 #3327 Recover the dropped attachments from retained maildir .eml (ADR-145 D6) restores โ‰ฅ71,440 dropped documents All

Blocked by gates G0 (#3323 โ€” closed), G1 (disk), G2 (writers), G3 (unit cost), G4 (#3325). Nothing is lost: every maildir message's raw .eml is retained โ€” bucket documents, prefix */maildir/* โ€” 273,648 objects / 190 GB, oldest 2026-06-16, and maildir ingestion started 2026-06-16 so all 53,166 maildir messages fall inside it. No vendor refetch, no re-ingest, no retention clock โ€” this optimizes for correctness, not speed. Recoverable floor: โ‰ฅ71,440 documents, corpus-scale โ€” plausibly the same order as everything doc_extract has processed to date (65,629). Co-tracked with AMA-TRUST.

A watermarked, checkpointed pass over documents/*/maildir/*: re-parse the .eml, upload attachments 2..N, insert message_attachments rows against existing messages rows, let the existing fan-out do the rest. The doc_extract fan-out needs no re-shaping โ€” it is already per-attachment (dedup_key=attachment_storage_key) and only_attachment_keys is already a set; only the enumeration feeding it was capped.

Two things DO break on N and are fixed here: _is_already_extracted resolves the node by msg.media_filename โ€” a message-level scalar โ€” and must resolve per attachment; and the bounded-attempt budget is watermarked on the message row (_resolve_outcome_row), so one poison file parks its healthy siblings โ€” it must move to the attachment. storage_key_to_message_id is built 1:1 (enrich_context_seeding.py:161-164); doc_extract_drain._unit_target keys by (source_id, storage_key).

Scope limit โ€” maildir only. Only maildir persists raw RFC822; Exchange uploads no raw, Gmail decodes in memory and discards, Slack/WhatsApp store individual attachments only. Other lanes get the model fix (no future loss) but their historical loss needs a vendor refetch โ€” out of scope.

Gate G1 is recurring, not one-time. Disk headroom must be re-read from the Supabase dashboard during the pass with a stated abort threshold โ€” autoscale is capped at 4 expansions/24h and the 95% failure mode is read-only, and if the instance goes read-only mid-batch the checkpoint write is also a Postgres write, so the checkpoint must be idempotent under a failed mid-write batch.

Acceptance โ€” Regina's invoice: ..._Sum.xlsx (34,515 B) landed, ..._Invoice.pdf (315,799 B) is unstored; both decodable today from gm-inlined-rfc822-backfill/inlined-originals.jsonl. The invoice must be retrievable by its content, not just structurally present.

Ordering

Why is it in this order?

The four defects are a chain, not a menu โ€” each rung breaks ties left by the one above it.

  • Stop the silent loss first. The substrate model (the message_attachments issue) is the root cause: no connector patch can fix it, and 57% of document-bearing Regina maildir messages lost at least one. It comes first because it stops the ongoing loss โ€” recovery only rescues history.
  • Rule the audience before the population grows. The recovery multiplies cross-message dedup, so the disclosure decision must ship before the backfill โ€” otherwise the defect's blast radius grows under it.
  • Make it visible, then recover. The serving cap and the recovery ship together: recovered documents are invisible without the serving fix, and the recovery is corpus-scale and correctness-over-speed.

What this order deliberately does NOT use:

  • Issue age or filing order โ€” all four were filed within minutes on 2026-08-15.
  • Priority labels alone โ€” the one p2 is the largest item (โ‰ฅ71,440 documents), and the p1s are not equally urgent.
  • How cheap something looks โ€” the root cause cannot be fixed by any connector patch, and the serving fix touches ten live objects.
  • Severity labels โ€” the track's rank is by impact, not by priority chips.

Standing rulings

History

The gate that unblocked the model โ€” the identity choke-point

The track is the attachment-model cluster (ADR-145), filed 2026-08-15 and pinned at rank 2: four open defects spanning message storage, audience scope, the serving cap, and recovery of what was already dropped. It has never been published.

Every other stage was gated by the identity choke-point (#3323, ADR-145 D3, gate G0), which closed 2026-08-15. Six independent expressions answered one question โ€” 'is this document already here?' โ€” across four keys (content_hash, filename, storage_key, message row), coherent only because there is one attachment per message. The fix collapsed them onto one content_hash-keyed resolver, deleted the filename/storage_key/message-row sites rather than repointing them (repointing preserves the two-different-invoice.pdf-are-one-document filename collision), collapsed version predicates onto is_current_extractor_version, and shipped a check.py AST hook in the same PR.

Its constraint (panel finding F2) still binds the recovery: content_hash needs the bytes, so without a cheap byte-free negative check in front of the resolver an already-extracted attachment must be downloaded to decide to skip it.