> ## Documentation Index
> Fetch the complete documentation index at: https://aidocs.ethanbragdon.icu/llms.txt
> Use this file to discover all available pages before exploring further.

# FORGE Operations Runbook

> Operate Starfire AI FORGE queues, builds, validation, repair loops, artifacts, workers, and systemic failures from Control Center.

# FORGE operations runbook

Use the FORGE operations view to distinguish one bad build from a systemic worker, provider, validation, or storage problem.

## 1. Find the build

Search by build ID, user, organization, project, or time window when the current Control Center surface supports those fields.

## 2. Identify the current/failed stage

Check whether the build is:

* queued
* planning
* preparing workspace
* generating
* validating
* repairing
* packaging
* completed/failed/cancelled

## 3. Check queue and worker health

If many unrelated builds are stuck in `QUEUED` or the same processing stage, inspect worker/queue health before retrying individual jobs.

## 4. Check provider/model health

If planning/generation calls fail broadly, correlate with model/provider metrics and routing.

## 5. Check validation

For build-specific validation failures, inspect validator output and repair attempts. Repeated identical failures often indicate a requirement, dependency, or environment issue rather than temporary infrastructure.

## 6. Check packaging/storage

A build can generate and validate successfully but fail during artifact packaging or upload. Inspect artifact/storage state separately.

## 7. Cancel or retry

Cancel when the build should no longer consume resources. Retry/requeue only after identifying whether the failure was temporary and whether source context is still current.

## 8. Preserve historical truth

Do not mutate a failed build record into “completed.” A retry should preserve the original failure and create a new attempt/resource according to the active job model.

## 9. Escalate systemic failures

Create/update an incident when many builds are affected by the same queue, worker, provider, validator, or storage problem.

<Warning>
  Operators should use build metadata and logs to diagnose jobs. Routine FORGE operations should not require unrestricted browsing of private source contents or artifacts.
</Warning>