> ## Documentation Index
> Fetch the complete documentation index at: https://aidocs.ethanbragdon.icu/llms.txt
> Use this file to discover all available pages before exploring further.

# Test & Evaluate Agents

> Evaluate Starfire AI agents against normal, ambiguous, adversarial, missing-context, tool, and permission cases before broader use.

# Test & evaluate agents

An agent that performs well on one ideal prompt is not ready for a workflow. Test the behavior you expect under normal and failure conditions.

## Build a small evaluation set

Include at least:

1. a normal task the agent should complete
2. an ambiguous task that should trigger clarification or careful assumptions
3. a task with missing evidence
4. a request that conflicts with the agent's constraints
5. a task requiring a tool the agent does not have
6. a task that should be refused or escalated because of permission/approval boundaries

## Evaluate more than writing quality

Check whether the agent:

* follows the durable role
* preserves required constraints
* uses tools when evidence is needed
* distinguishes facts from assumptions
* does not claim validators or external actions ran when they did not
* stays inside Project/Knowledge scope
* handles unavailable capabilities clearly

## Tool tests

For tool-backed agents, test tool failures too. The agent should not convert a failed search, repository lookup, validator, or external action into a confident success claim.

## Permission tests

Use a test account/context with narrower permissions to verify the agent does not depend on accidental administrator access.

## Regression testing

After changing agent instructions, tools, model eligibility, or Knowledge sources, rerun the same evaluation set. A change that improves one scenario can break another.

## Automation readiness

Before attaching an agent to a schedule, watcher, or multi-step workflow, verify that it behaves safely without constant conversational correction.

<Tip>
  Treat an important agent like a small software component: define expected behavior, test it, change one thing at a time, and rerun the checks.
</Tip>