​

Provider health runbook

Use this runbook when model, search, embedding, or other upstream provider behavior appears degraded.

​

1. Confirm scope

Determine whether the issue affects:

• one model
• several models on one provider
• one route/region
• one workload such as streaming or tools
• every provider

A single-user failure is not enough to declare a provider outage.

​

2. Review provider metrics

Inspect available signals such as:

• request volume
• success rate
• latency
• timeout rate
• rate-limit responses
• provider server errors
• token/request throughput
• fallback frequency

​

3. Compare routes

If the model has multiple routes, determine whether the problem is tied to one route or to the model/provider generally.

​

4. Check capability-specific failures

A provider can work for basic text while failing on streaming, tool calls, vision, or large contexts. Segment diagnostics by workload before disabling the entire provider.

​

5. Test a known request

Use an authorized diagnostic/test request rather than a user’s private prompt. Confirm expected response, latency, and capability behavior.

​

6. Decide mitigation

Possible mitigations include:

• remove an unhealthy route from normal traffic
• use a configured capability-compatible fallback
• temporarily reduce rollout
• disable one affected model
• wait for provider recovery while surfacing a clear user error

​

7. Create/update incident

When impact is broad or sustained, attach the provider/model components to an incident and record mitigation/recovery evidence.

​

8. Verify recovery

Do not restore broad traffic based on one successful request. Confirm error rate and latency return to acceptable behavior.