RECEIPTS PACK — STEP 7

Reliability Contract

What MemoryWell guarantees. What it tries to do. What it does when things go wrong.

Core Invariants (S1–S4)

Exactly-Once Write

Per idempotency_key per tenant, exactly one canonical Bottle is created. Concurrent storms, network retries, and duplicate submissions all produce a single canonical record.

↳ Single-writer lease election → nonce tie-breaking → loser back-off → BottleCatalog is_canonical=true

Tenant Isolation

No data belonging to Tenant A is ever readable or writable by Tenant B. Isolation is at the infrastructure query layer, not application logic.

↳ tenant_id derived from auth session only · every entity filter includes app_id=tenant_id · no payload-supplied tenant override

Audit Completeness

Every operation (store, search, recall, admin, delete) produces an AuditLog record. No operation can succeed silently. Content hash on every Bottle proves tamper-evidence.

↳ mwAuditLogger wired into all 5 core endpoints · SHA-256 content_hash computed on write · hash_chain_verified flag

Recoverability

Any system state can be exported and restored. No op is silently discarded. Outbox catches Redis drops. Repair sweep recovers orphans.

↳ mwExportSnapshot → import → count parity check · Outbox entity as durable fallback · mwReconcileOrphans for orphan recovery

Failure Modes & System Response

Failure ModeSystem ResponseGuarantee

Redis queue unavailableWrite to Outbox (durable DB fallback). Repair sweep re-queues on next cycle.No silent drop. May delay, not lose.

Worker crashes mid-flightJob status remains 'processing'. Health check detects stuck jobs. Re-queued with incremented attempt count.Retry up to max_attempts. Dead-letter after 4 failures.

Duplicate idempotency keyLease election: one winner creates the Bottle. All losers back off. Tombstone created for losers.Exactly 1 canonical Bottle. Zero duplicates.

Rate limit exceeded (429)Request rejected with 429 + retry-after header. Per-tenant RateLimitBucket enforced.No cross-tenant impact. Caller must retry after window.

Validation failure (bad payload)400 returned before queue entry. Job never created. Nothing to roll back.Fail-closed. No partial state.

Embedding service unavailableBottle stored without embedding. Added to EmbeddingBackfillQueue. Search degrades to keyword.Store succeeds. Search degrades gracefully. Backfill restores semantic search.

Database write timeoutOperation retried with exponential backoff + jitter. After max_attempts: dead-letter + AuditLog.No silent failure. Operator alerted via metrics.

GUARANTEED

✓ Exactly-once store per idempotency key (S1)
✓ Zero cross-tenant data leakage (S2)
✓ Every operation in AuditLog (S3)
✓ Full state recoverable from export snapshot (S4)
✓ No silent job drops (Outbox fallback)
✓ Tamper-evident content hash on every Bottle
✓ 429 returned before tenant cap is exceeded (never silently throttled)

BEST EFFORT

~ p95 store latency < 2s (depends on Base44 platform throughput)
~ Semantic search latency < 1.5s (depends on embedding model API)
~ Queue drain within 5 minutes (worker invoked on schedule, not demand)
~ Embedding backfill within 24h of store
~ Ordered recall of memories (approximate, not strict FIFO)