STAGE 10 — OBSERVABILITY & OPERATIONAL PROOF
Prove health. Detect drift. Fail loud. Recover clean.
Health Snapshot last 15m
Rate Limit
Allowed
0
Throttled
0
Fail-closed
0
Cost Guard
Allowed
0
Denied
0
Warn 80%
0
Idempotency
Won
0
Lost (replay)
0
Error
0
Outbox (DB live)
Pending
85
Processing
0
Done
44
Failed
0
Repair / Zombie
Orphan fixed
0
Zombie reset
0
Illegal state
0
Dependencies
Redis OK
0
Redis fail
0
Base44 fail
0
Invariant Violations Prevented
This is what makes it defensible
0
Duplicate writes prevented
by idempotency gate
0
Zombie jobs recovered
by zombie sweep
0
Orphan outboxes repaired
by repair sweep
0
Illegal transitions blocked
by state validator
Top Offenders
Top endpoints by throttles
None in window
Top job types by retries
None in window
Live Debug Stream
last 30 events
No events in window — run tests to populate
Stage 10 Red Team Suite (Tests G–K)
Test G
Redis Down → Fail ClosedSimulates Redis failure → expect 503 + dep.redis_fail metric
Test H
Base44 Throttle VisibilityBurst of outbox updates → expect retry metrics visible
Test I
Idempotency Collision20 concurrent same idem_key → expect 1 Memory, 19 claim_lost
Test J
Zombie Recovery VisibleStale processing job → zombie sweep → metrics emitted
Test K
Repair Sweep VisibleMemory in stored with no outbox → repair → metrics emitted
Stage 10 Pass Criteria
10.1 MetricEvent entity exists + append-only
10.2 Canonical metric keys in use (no ad-hoc)
10.3 All instrumentation points emitting (A–F)
10.4 Dashboard shows health + invariant counters + stream
10.5 Red Team Tests G–K all PASS
10.6 Failures never silent — every 503 emits metric