Documents and RAG

Documents power Rules Chat, campaign-history answers, monster extraction, NPC grounding, and some prep tools. The system supports admin core-rule documents and DM-uploaded campaign documents.

Backend

Primary files:

• server/app/routers/admin.py
• server/app/routers/campaign_documents.py
• server/app/routers/campaign_core_sources.py
• server/app/lib/core_rules_storage.py
• server/app/lib/campaign_docs_storage.py
• server/app/lib/rag/ingest.py
• server/app/lib/rag/extract.py
• server/app/lib/rag/chunking.py
• server/app/lib/rag/embeddings.py
• server/app/lib/rag/retrieve.py
• server/app/lib/rag/rerank.py
• server/app/lib/rag/monsters.py

Document Types

Type	Owner	Storage	Retrieval
Core rule	Platform admin	Private server/storage/	Required or campaign opt-in optional books.
Campaign document	Campaign DM	Campaign document storage	Only that campaign.

Core-rule usage controls retrieval:

• required: always a Sage source.
• optional: retrieved only when enabled for the campaign.
• tools: never retrieved, but still processed for structured data such as monsters.

Ingestion Pipeline

Ingestion runs:

source file -> extracted Markdown -> chunks -> embeddings -> database rows
           -> overview chunk -> monster extraction -> indexed status

The pipeline is best-effort for generated overview and monster extraction. Chunk persistence is the core success path.

Process vs Re-index

Re-index parses the source file again. Process reuses saved extracted Markdown and re-derives chunks, overview, and monsters. Use Process when chunk settings or downstream extractors changed but source parsing does not need to run again.

Retrieval Pipeline

Retrieval combines query alias expansion, dense vector search, Postgres full-text search, Reciprocal Rank Fusion, reranking, and deterministic overview-chunk inclusion.

The generator receives full chunk contents, while citations use shorter snippets for display.

Change Checklist

• Keep source storage private/public according to document type.
• Preserve campaign and core-rule retrieval scope.
• Do not let optional add-ons fail the whole indexing run.
• Avoid blocking the event loop during parsing, embedding, or model calls.
• Update ingestion settings UI and defaults together.