> ## Documentation Index
> Fetch the complete documentation index at: https://aidocs.ethanbragdon.icu/llms.txt
> Use this file to discover all available pages before exploring further.

# Incident Response Runbook

> Detect, scope, mitigate, communicate, monitor, resolve, and review Starfire AI platform incidents using component health and timelines.

# Incident response runbook

Use this runbook for platform-impacting incidents rather than isolated support cases.

## 1. Detect and confirm

Use health checks, error trends, provider metrics, support reports, and request/run failures to confirm the impact is broader than one account.

## 2. Create the incident

Record:

* title
* affected components
* initial impact
* detection time
* status `INVESTIGATING`
* current evidence

Avoid declaring a root cause before it is supported by evidence.

## 3. Scope the blast radius

Determine:

* user populations affected
* organizations/regions/deployments affected
* model/provider scope
* web vs Developer Platform
* background jobs affected
* billing/auth/storage dependencies involved

## 4. Mitigate

Choose the smallest action that reduces user impact safely:

* disable an unhealthy route
* use capability-compatible fallback
* pause a failing feature
* stop/restart affected workers according to operational procedures
* enable maintenance when necessary
* roll back a recent configuration or release when supported

Record each significant mitigation in the timeline.

## 5. Communicate

Use factual updates: what is affected, what operators are doing, and what changed since the previous update. Do not publish unverified internal theories as confirmed root cause.

## 6. Monitor recovery

After mitigation, move to `MONITORING` only when service indicators show recovery. Watch error rate, latency, queues, provider health, and representative workflows.

## 7. Resolve

Mark `RESOLVED` only after recovery has been verified—not merely because a fix was deployed.

## 8. Follow-up

For significant incidents, capture:

* confirmed root cause
* detection gap
* mitigation effectiveness
* rollback/recovery behavior
* customer impact window
* prevention/follow-up actions

## Timeline discipline

Preserve the timeline as an operational record. Do not rewrite old entries to make early uncertainty disappear; add later confirmed findings instead.

<Card title="Incidents" icon="triangle-exclamation" href="/admin/incidents">Review incident states, components, and timeline-writing guidance.</Card>