Skip to main content

​
Knowledge source ingestion

A Knowledge source is not ready for semantic retrieval the instant it appears in storage. Starfire processes durable sources through an ingestion pipeline.

​
Ingestion lifecycle

​
Add a source

  1. Open the Knowledge or project Knowledge surface.
  2. Choose the owning personal, project, or organization context.
  3. Add a supported file, repository, URL, or other available source type.
  4. Wait for processing/indexing state to complete.
  5. Test retrieval with a query that should clearly match the source.

​
Extraction

Extraction converts the original source into material Starfire can index. A source can fail here because the format is unsupported, the content is corrupt, or the extractor cannot read useful text.

​
Chunking

Large content is divided into retrievable units. Chunk metadata should preserve enough source location and identity for Starfire to connect retrieved material back to its source.

​
Embeddings and indexing

Configured embedding/index services turn chunks into searchable representations. Provider health or indexing-worker health can delay this stage even when the upload itself succeeded.

​
Ready state

A source should be considered reliable for Knowledge queries only after the relevant indexing state is ready.
Do not diagnose a retrieval problem by repeatedly uploading duplicate copies of the same source. First inspect whether the original source failed extraction, indexing, permissions, or retrieval.

RAG & retrieval

See how indexed content is selected at query time.