Skip to main content

​
Incident response runbook

Use this runbook for platform-impacting incidents rather than isolated support cases.

​
1. Detect and confirm

Use health checks, error trends, provider metrics, support reports, and request/run failures to confirm the impact is broader than one account.

​
2. Create the incident

Record:
  • title
  • affected components
  • initial impact
  • detection time
  • status INVESTIGATING
  • current evidence
Avoid declaring a root cause before it is supported by evidence.

​
3. Scope the blast radius

Determine:
  • user populations affected
  • organizations/regions/deployments affected
  • model/provider scope
  • web vs Developer Platform
  • background jobs affected
  • billing/auth/storage dependencies involved

​
4. Mitigate

Choose the smallest action that reduces user impact safely:
  • disable an unhealthy route
  • use capability-compatible fallback
  • pause a failing feature
  • stop/restart affected workers according to operational procedures
  • enable maintenance when necessary
  • roll back a recent configuration or release when supported
Record each significant mitigation in the timeline.

​
5. Communicate

Use factual updates: what is affected, what operators are doing, and what changed since the previous update. Do not publish unverified internal theories as confirmed root cause.

​
6. Monitor recovery

After mitigation, move to MONITORING only when service indicators show recovery. Watch error rate, latency, queues, provider health, and representative workflows.

​
7. Resolve

Mark RESOLVED only after recovery has been verified—not merely because a fix was deployed.

​
8. Follow-up

For significant incidents, capture:
  • confirmed root cause
  • detection gap
  • mitigation effectiveness
  • rollback/recovery behavior
  • customer impact window
  • prevention/follow-up actions

​
Timeline discipline

Preserve the timeline as an operational record. Do not rewrite old entries to make early uncertainty disappear; add later confirmed findings instead.

Incidents

Review incident states, components, and timeline-writing guidance.