STAGE 9 — TRANSACTIONAL MEMORY STORE

Integrity Dashboard

Convergent-Once · Crash-Safe · Idempotent · Observable

🔬 Platform Boundary Discovered (Sage — Feb 25, 2026)

Base44 uniqueConstraints are advisory only — not DB-enforced. Under true concurrency, multiple records with the same idem_key or dedupe_key can be created. This means hard exactly-once requires an external atomic primitive (Cloudflare Durable Object, Upstash Redis, or Postgres ON CONFLICT).

Current contract: "Convergent-Once" — advisory lock via StoreOperation (announce → 150ms settle → earliest-wins) guarantees convergence under test conditions. Production-grade hard exactly-once is Stage 9 Option 1 (external gatekeeper).

✅ Current: Convergent-Once (Base44-native)🚀 Option 1: Hard Exactly-Once (DO/Redis)

Core Invariants

Canonical IdempotencyMultiple writes may land, but exactly one becomes canonical. Others are replays — all resolve to same memory_id.

Outbox ConvergenceDuplicate jobs may be created under burst, but only one executes. Dupe Sweep cancels non-canonical jobs.

No Orphan MemoryIf Memory in stored with no Outbox → repair sweep creates missing job and transitions state.

No Lost WorkAll Outbox jobs eventually done/failed/retry — zombie detector resets stale claims after 5 min.

Idempotent EffectsExtract + embed may run multiple times but outcome is identical — written by (memory_id, version) key.

Memory State (115 total)

stored0

pending_extract114

extracted1

pending_embed0

ready0

failed0

Outbox State (129 total)

pending85

processing0

done44

failed0

Integrity Controls

Red Team Attack Suite

Test A — Double SubmitI1I2I5

20 simultaneous requests with same idem_key. Advisory lock: 1 winner creates Memory+Outbox, 19 get replay. Expect: 1 Memory, 1 extract, 1 embed.

Test B — Crash After BottleI1I3

Bottle created, crash before Memory. Retry same idem_key. Expect: exactly 1 Memory in correct state.

Test C — Crash After MemoryI3

Memory exists in stored with no Outbox. Repair sweep must create missing job + transition state.

Test D — Worker Crash Mid ExtractI4

Job claims, crashes (zombie >5 min). Zombie sweep detects stale claim and resets to pending.

Test E — Embed Duplicate ProtectionI2I5

10 simultaneous embed job attempts. Advisory lock: 1 winner creates, 9 skip. Exactly 1 in DB.

Test F — Outbox FloodI4

100 pending jobs created and processed. None stuck, no double transitions.

Memory State Machine (legal transitions only)

stored

→

pending_extract

→

extracted

→

pending_embed

→

ready

Illegal transitions (skip, regression, circular) → 500 error

Execution Log

Waiting for activity...

Stage 9 PASS Criteria

Contract: Convergent-Once (Base44-native advisory lock pattern)

❌All 6 red-team tests pass

❌I1: 1 canonical winner per idem_key race

❌I4: No stuck jobs / zombies

❌I3: No orphan Memory (stored + no Outbox)

❌I2: Outbox converges to 1 per event type

❌Zero integrity violations

Stage 9 Option 1 — Hard Exactly-Once Roadmap

Sage's recommendation: add one external atomic primitive. Keep Base44 for everything else.

9.1

Add Gatekeeper APIfuture

POST /claim/idempotency · POST /claim/outbox · POST /lease/job

Cloudflare Durable Object or Upstash Redis (SETNX)

9.2

Wire mwStoreMemoryTransactionalfuture

claimIdempotency(tenant, idem_key) before Memory create. If lost → return existing.

2–3 lines of code in mwStoreMemoryTransactional

9.3

Wire outboxWorker claimsfuture

leaseJob(outbox_id) before execute. Real atomic lease with expiry. No more best-effort claimed_by.

Replace current claimed_by pattern in outboxWorker

9.4

Update red-team suitefuture

Tests A+E expect exact counts (0 duplicates), not convergence. Crash tests become stronger.

Remove advisory lock fallback logic from tests