Mindbridge Β· track status

DEPLOY-SAFETY

The serving tier and the background work must not fight over the same machines, and a deploy must not change what the product does except by what the diff says. Born 2026-08-18 out of the migrations-vs-deploy gate defect, and named a track by usage when the maildir-poll cutover needed a label β€” this page is its first, written 2026-08-25.

Label used from 2026-08-18 (the deploy-gate fix) Β· page created 2026-08-25 during the 5-day track reconciliation Β· figures quoted from the PR bodies' own production measurements.

Next β†’ Part 2 of the maildir-poll cutover β€” the migration-only follow-up that retires the cron endpoint still driving the poll. Part 1 (PR #3550, merged 2026-08-23) registered the poll as a maildir_poll drain loop on the worker and moved the app tier to dedicated CPU, but it has no production effect on the poll's old path until part 2 lands β€” the cron keeps firing the endpoint, so today BOTH drivers run and new mail can surface at tick + 3600s β‰ˆ 70 minutes, not hourly. #3549 carries state:not-in-effect until part 2 ships.
3 open β€” and the one merged item in this label's history already paid for itself twice over: the app tier was stealing 7.54% CPU on the shared 4-worker box (versus 0.007% on a dedicated-2x machine β€” a 1,032Γ— difference), and that steal, not the maildir walk, was what stalled AMA turns. Post-move steal reads 0.09%. The same PR's measurement work seeded the AMA turn-time instrumentation on ama_turns (PR #3552, on AMA-TRUST).
3open issues
7.54% β†’ 0.09%app-tier CPU steal, before β†’ after
302 / 96,114mailboxes / messages walked per ~10-min tick
~70 minreal poll period until part 2
Open work Why this order History

Open work

The maildir-poll cutover

p1 #3549 Maildir live poll runs inline on the serving tier (2-part cutover) part 1 merged β€” part 2 (retire the cron endpoint) owed All
Part 1 shipped 2026-08-23 (PR #3550): the hourly poll is registered as a maildir_poll drain loop on the worker machine, and the app tier moved onto performance-2x dedicated CPU. πŸ›‘ No production effect until part 2: the cron endpoint keeps driving the poll β€” a migration-only follow-up retires it, and until then the change is the placement (worker + dedicated CPU), not the scheduling. Honest boundary, stated by the panel: an earlier draft blamed the maildir walk for the AMA stalls; that causal claim was refuted and retracted β€” the steal was the cause, the walk merely shared the box. The real poll period is tick + 3600s β‰ˆ 70 minutes, so new mail can surface later than it did today; that is a documented trade, not an accident.
Why is it in this order?

One item, one order. The track's only open issue is the second half of a cutover whose first half is live β€” finishing it removes the duplicated driver and the 70-minute period ambiguity in one migration. Anything else that lands on this label later (a deploy-gate defect, a machine-placement regression) enters by the same criterion this page's siblings use: what a user experiences first.

What the audit found β€” generated

Evidence below is generated from the read-only audit of 2026-08-29 and reconciled against the open issues at build time. Every claim cites the issue it came from. Nothing here is a decision β€” rulings stay in the prose above, which is written by hand.

#1591 Auto-restart mawja-api machines on sustained health-critical (#1542 option 3)absencehigh confidence

Detection exists and is armed; automatic recovery does not. `.github/workflows/fly-health-alert.yml` (cron `*/5 * * * *`) runs `docs/reference/operations/alerting/fly-health-check-alert.sh` over the matrix mawja-api / mindbridge-evolution / mindbridge-evolution-db β€” it alerts only. Search for any restart action: `git grep -rn "machine restart|/restart|machines/.*restart" origin/main -- .github docs/reference/operations backend/scripts` β€” population = the entire CI-workflow tree, the ops docs tree, and backend scripts on origin/main. Six hits, ALL prose in runbooks (docs/reference/operations/monitoring.md:227, 243, 247, 306, and 322 which explicitly names this as the missing complement) plus one line in the DeepSeek re-enrich runbook. Zero code path calls the Machines API restart endpoint.

#3549 [deploy-safety] Maildir live poll runs inline on the serving tier (2-part cutover)prodhigh confidence

Part 1 shipped and is reachable: `maildir_poll` is a registered drain loop at `backend/app/worker.py:220` (named in the loop roster docstring at line 157). Part 2 has NOT shipped: `git grep -rn "poll-maildir-tenants" origin/main -- supabase/migrations` returns only `20260616174657_pgcron_poll_maildir_tenants.sql` (the original SCHEDULE migration; its `cron.unschedule` is the idempotent pre-schedule guard inside that same file) and a comment line in `20260817134432_schedule_cron_delivery_sweep.sql`. No unschedule migration exists. Prod cron census 2026-08-29: jobid 18 `poll-maildir-tenants`, `0 * * * *`, active, POST /api/connectors/internal/poll-maildir, last start 13:00:00 succeeded, 24 runs/24h. The endpoint is still driving the walk on the serving tier.

Open questions β€” what the audit could not settle
  • #1591 Whether the detection leg actually fires end-to-end (the script exits 0 and skips when FLY_HEALTH_SLACK_WEBHOOK is unset). `gh secret list` shows FLY_API_TOKEN and FLY_API_TOKEN_READONLY but I did not confirm FLY_HEALTH_SLACK_WEBHOOK. Close with: `gh secret list | grep FLY_HEALTH_SLACK_WEBHOOK`, and `gh run view --log $(gh run list -w fly-health-alert -L1 --json databaseId -q '.[0].databaseId')` to see whether the poll skipped or ran.
  • #3549 Whether a maildir poll has actually been observed executing on the worker machine (the label-removal precondition). Close with: `python3 .claude/tools/flylogs/flylogs.py mawja-api --since 6h --grep maildir_poll --context 5`.
Not yet assessed β€” filed after the audit
  • #3946 [deploy-safety] a status-board label error turns a green production deploy red β€” decouple…

3 assessed issues closed since the audit and are not shown β€” a closed issue's evidence is history, and printing it as current is how a page starts lying.

History

The label's first use β€” the deploy gate (2026-08-18)

PR #3453 (merged 2026-08-19; the label itself was first used 2026-08-18 on #3442) gated the backend deploy on the DATABASE's schema rather than the migrations job's conclusion β€” closing the class where a migration job reports success while the schema the app deploys against is not what the migrations produced (#3442). That fix predates this page; it is why the label exists.

The measurement that justified the CPU move (2026-08-23)

One maildir tick walked 302 mailboxes (96,114 messages deduped) over ~10 minutes β€” inline, on the serving tier. The steal table across machines read 7.54% (shared 4-worker box) versus 0.007% (dedicated) β€” a 1,032Γ— difference β€” and tool calls measured 36–63s with the DB idle at 1–3s against 1.8–4.0s on the healthy comparison, which is the signature of CPU starvation rather than query cost. After the move, steal read 0.09% β€” and the follow-on turn-time instrumentation found a Regina turn still costs ~2.25Γ— an aeldaly turn at the same LLM round count, with ~30s/turn still unattributed: the steal was real but not the whole story, and the rest belongs to AMA-TRUST.