Skip to content

Memory Orchestration Monitoring Specification (SLO/SLA)

Last updated: 2026-08-22

1. Objective

This specification defines production monitoring metrics, SLO/SLA targets, alert conditions, and operational response criteria for MemoryOrchestrator.

2. Scope

  • request normalization
  • memory retrieval
  • model execution
  • learning update
  • memory writeback
  • degraded / fail-fast / fallback snapshot behavior

3. Key metrics

3.1 Latency

  • orchestrator_request_latency_ms (histogram)
  • labels: status, source, mode
  • SLO:
  • p95 < 800 ms
  • p99 < 1500 ms

3.2 Success and degradation ratios

  • orchestrator_requests_total (counter)
  • labels: status=completed|degraded|failed
  • orchestrator_degraded_total (counter)
  • SLO:
  • completed ratio >= 99.0% (rolling 7d)
  • degraded ratio <= 0.8% (rolling 1d)

3.3 Stage-level failures

  • orchestrator_stage_failures_total (counter)
  • labels: stage, error_code, retryable
  • SLO:
  • MLI-RET-001 <= 0.3%
  • MLI-EXE-001 <= 0.2%
  • MLI-WRB-001 <= 0.2%

3.4 Fallback snapshot reliability

  • orchestrator_fallback_snapshot_total (counter)
  • labels: failed_stage, error_code
  • orchestrator_fallback_snapshot_persist_fail_total (counter)
  • SLO:
  • snapshot persistence success >= 99.9%

3.5 Causal extraction quality

  • orchestrator_causal_context_items (histogram)
  • orchestrator_causal_edges_items (histogram)
  • Quality guardrails:
  • neutral workload: causal_edges p95 <= 0
  • causal workload: causal_edges p50 >= 1

4. SLA

  • Availability SLA: 99.5% / month
  • Incident response:
  • Sev1 acknowledgement <= 15 min
  • Sev2 acknowledgement <= 60 min
  • Recovery target:
  • degraded surge containment <= 2 h

5. Alert rules

  1. Critical
  2. Condition: failed ratio > 2% over 5 minutes

  3. Action: prioritize fallback path and isolate fail-fast dependent flows

  4. High

  5. Condition: sustained MLI-RET-001 threshold breach for 10 minutes

  6. Action: inspect retrieval backend health, cache/index sync state

  7. High

  8. Condition: orchestrator_fallback_snapshot_persist_fail_total > 0
  9. Action: inspect storage permissions, capacity, filesystem state

6. Operational response flow

  1. Alert fired
  2. Confirm failed/degraded ratios
  3. Break down by stage/error_code
  4. Correlate fallback snapshots and audit trail
  5. Apply rollback or stronger safe mode
  6. Run postmortem and tune thresholds

7. References

  • MEMORY_ORCHESTRATION_E2E_SPEC.en.md
  • EvoSpikeNet-Core/specs/system_orchestration_spec.md