AI Knowledge Assistant (RAG) Setup

How to Use This Template

Answer the questions in each section. Replace every `[[Token]]` with your real value. Where you see a table, add or remove rows to match your actual document library. The result is a working RAG architecture plan you can hand to a developer, run through a no-code platform yourself, or use to configure an AI agent.

---

Section 1 - Use Case Definition

Before touching any tooling, define what your assistant must do and for whom. Vague inputs produce vague retrieval.

Audience: [[Who will query this assistant - e.g., "Internal support team", "Customers on a public portal", "Analysts in a data team"]]

Primary use case: [[One sentence - e.g., "Answer customer support questions about our product using our help docs and policy PDFs"]]

Query style: [[How users will ask - e.g., "Short keyword questions", "Long conversational questions", "SQL-style structured requests"]]

Languages supported: [[e.g., "English only" / "English + Spanish"]]

Acceptable answer format: [[e.g., "2–4 sentence summaries with a source link" / "Step-by-step instructions" / "Bullet list with page references"]]

Out-of-scope topics (the assistant should refuse): [[e.g., "Legal advice", "Medical diagnosis", "Competitor comparisons"]]

---

Section 2 - Document Inventory and Ingestion Strategy

List every document type in scope. This drives chunking, metadata design, and re-index cadence.

Total estimated document count: [[e.g., "~1,000 pages across 150 files"]]

Languages in the corpus: [[e.g., "English only" / "English + French"]]

Documents to exclude from indexing: [[e.g., "Draft files, files marked CONFIDENTIAL-LEGAL, files older than 3 years"]]

Ingestion pipeline

The ingestion pipeline runs before any query reaches the assistant. Define it once, automate it.

Step 1 - Extract: Convert raw files to clean text.

• PDF: use a PDF parser (PyMuPDF, AWS Textract for scanned docs, or Unstructured.io for mixed-format batches). Do not rely on copy-paste - OCR quality matters for retrieval.
• HTML/Web pages: strip nav, footer, ads; keep body text and headings.
• DOCX/Slides: extract text + speaker notes; discard decorative images unless they contain text.
• CSV/JSON: flatten structured rows into prose snippets or keep as structured chunks (see Section 3).

Step 2 - Clean: Remove boilerplate that adds noise without adding meaning - page numbers, headers/footers repeated on every page, legal fine print that appears in every contract, watermarks.

Step 3 - Metadata tagging: Tag every document at ingest with at minimum:

---

Section 3 - Chunking Strategy

Chunking is the highest-leverage decision in RAG. Too large → irrelevant context floods the LLM. Too small → answers lose context and coherence.

Recommended chunk sizes by document type

Your primary document type: [[Select from table above]]

Your target chunk size: [[e.g., "600 tokens with 100-token overlap"]]

Chunking method

Recursive text splitting (default for mixed content): split first on double-newline (paragraphs), then on single newline, then on sentence boundaries, then on space - never mid-word. This preserves natural reading units better than fixed-size splitting.

Semantic chunking (higher quality, higher compute): split when embedding similarity drops below a threshold between consecutive sentences. Produces more coherent chunks but requires an extra embedding pass at index time. Recommended for high-value knowledge bases where retrieval precision matters.

Agentic chunking (for very dense technical content): pass each section to an LLM and ask it to produce one proposition per chunk. Expensive - use only for high-value domains (e.g., legal contracts, scientific papers).

Your chosen chunking method: [[Recursive text splitting / Semantic / Agentic]]

Chunk metadata injection

Prepend a context header to every chunk so the embedding captures document-level context, not just local text:

```
[Source: [[Document Title]] | Type: [[Document Type]] | Date: [[Date Published]]]
[[Full chunk text here]]
```

This dramatically improves retrieval accuracy for queries that reference a specific document type or time period.

---

Section 4 - Embedding Model and Vector Store

Embedding model selection

Your embedding model: [[Select from table above]]

Reason for choice: [[e.g., "English-only corpus, cost-sensitive → text-embedding-3-small"]]

Vector store selection

Your vector store: [[Select from table above]]

Deployment model: [[Managed cloud / Self-hosted / Local (dev only)]]

Index type: Use HNSW (Hierarchical Navigable Small World) for all production deployments - it gives near-exact nearest-neighbor search at sub-millisecond latency without scanning the full index.

---

Section 5 - Retrieval Approach

Top-k retrieval baseline

The simplest retrieval: embed the query, find the `k` most similar chunks, pass them to the LLM.

Your top-k value: [[e.g., "k=5 for fast responses; k=10 for complex multi-part questions"]]

A top-k of 5–8 covers most use cases. Larger k means more context for the LLM but higher cost and risk of irrelevant chunks diluting the answer.

Hybrid retrieval (recommended for production)

Pure semantic search misses exact keyword matches (product codes, names, dates, technical terms). Hybrid combines:

• Semantic (vector) search - finds conceptually similar chunks even when phrasing differs.
• Keyword (BM25/full-text) search - finds exact term matches; critical for model numbers, proper nouns, dates.

Merge results using Reciprocal Rank Fusion (RRF): assign rank scores to results from each method independently, then combine them. No manual weight-tuning required.

Hybrid retrieval enabled: [[Yes / No - if No, state why semantic-only is sufficient]]

Keyword search method: [[BM25 / Elasticsearch / Typesense / pgvector full-text]]

Metadata filtering (pre-retrieval)

Before semantic search runs, filter by metadata to narrow the candidate pool:

Reranking (strongly recommended)

Top-k retrieval returns the most *similar* chunks, not necessarily the most *relevant*. A reranker is a cross-encoder model that scores each (query, chunk) pair directly - far more accurate than cosine similarity alone, but too slow to run on the full corpus.

Reranking pipeline:

1. Retrieve top-20 candidates from hybrid search.

2. Pass all 20 (query + chunk) pairs to the reranker.

3. Reranker scores each pair; keep the top-5.

4. Pass top-5 to the LLM for generation.

Your reranker: [[Select from table above / None - explain tradeoff]]

Retrieve top-N before reranking: [[e.g., "20"]]

Final top-k passed to LLM after reranking: [[e.g., "5"]]

---

Section 6 - LLM / Generation Layer with Grounding and Citations

Model selection for generation

Your generation model: [[Select from table above]]

System prompt for grounding

The system prompt is the primary hallucination control. Use this template:

```
You are [[Assistant Name]], a knowledge assistant for [[Organization Name]].
Answer ONLY using the context passages provided below.
If the answer is not in the context, say: "I don't have that information in my knowledge base."
Do NOT make up facts, statistics, or policy details.
Always cite the source of your answer using the format: [Source: [[Document Title]], [[Page or Section]]].
Keep answers concise - 2–4 sentences unless the user asks for more detail.
Today's date is [[Current Date]].
```

Your assistant name: [[e.g., "AskAcme" / "HR Helper" / "PolicyBot"]]

Tone: [[e.g., "Professional and concise" / "Friendly and plain-language" / "Technical"]]

Citation format

Every answer must include one or more citations. Define the format:

Citation template: [[e.g., `[Source: {documenttitle}, {sectionname}, p.{pagenumber}]` / `[Ref: {sourceurl}]`]]

Source link behavior: [[Always show source URL / Show URL on request / Show document title only]]

"I don't know" trigger: The LLM must say "I don't have that information" - never guess - when retrieved chunks don't contain a clear answer. Include this instruction verbatim in the system prompt.

Context window management

When multiple chunks are passed, order matters:

1. Place the most relevant chunk (top reranker score) first.

2. Place the second-most relevant chunk last (the "lost in the middle" effect - LLMs weight beginning and end of context more than the middle).

3. Separate chunks with a clear delimiter: `--- [Source: {title}] ---`

Max chunks passed to LLM: [[e.g., "5"]]

Estimated tokens per chunk: [[e.g., "600 tokens"]]

Estimated system prompt tokens: [[e.g., "300 tokens"]]

Total context per query: [[Calculate: (maxchunks × tokensperchunk) + systemprompt + user_query ≤ model context window]]

---

Section 7 - Chat Interface and Source Links

UI / tool options

Your chosen interface: [[Select from table above]]

Source link format: [[Show as clickable URL / Show as "Document: title" text / Show in a collapsible references panel]]

Feedback button (thumbs up/down): [[Yes - feeds eval loop / No]]

---

Section 8 - Access Control and Data Privacy

Access control in RAG has two layers: who can query the assistant, and which documents a user is allowed to see in retrieval results.

Query-level access control

Your access control method: [[Select from table above]]

Document-level access control (row-level security)

When different users should see different subsets of documents, enforce it at retrieval time - not at the UI level.

Implementation: Store `accesstier` (or `usergroups`) as metadata on every chunk. At query time, pass the authenticated user's access level as a metadata filter. The vector store applies the filter before returning any results.

Your user tiers: [[e.g., "public / employee / manager / hr-only / legal-only"]]

Filter field name: [[e.g., `access_tier`]]

User group mapping:

Data privacy rules

Does your corpus contain PII (names, emails, employee records)? [[Yes / No]]

If yes:

1. PII fields are metadata - do not include raw PII inside embeddable chunk text unless the user querying that chunk is authorized to see it.

2. Confirm your embedding provider's data processing agreement (DPA) covers your data type. OpenAI, Cohere, and Anthropic all have DPAs - check their trust pages.

3. For GDPR / HIPAA / CCPA: use a self-hosted embedding model and vector store to keep data on-premises.

4. Audit log: log every query with timestamp, user ID, and chunks retrieved (not the chunk text) - keep logs for [[30 / 90 / 365]] days per your compliance policy.

Self-hosted required for compliance? [[Yes - list reason / No]]

Your embedding and vector store deployment: [[Managed cloud / Self-hosted / Hybrid]]

---

Section 9 - Evaluation Framework

Evaluation is not optional - it is the only way to know if the assistant is reliable enough to deploy and whether it degrades over time.

The four metrics to track

Evaluation tooling

Your primary eval tool: [[Select from table above]]

Test set construction

Build a test set of at least 50 question-answer pairs before launch:

1. Write 50 questions representative of real user queries.

2. For each question, identify the ground-truth chunk(s) that contain the correct answer.

3. For each question, write the correct answer as a human would.

4. Run the full RAG pipeline on each question and score against ground truth.

Test set size: [[e.g., "50 questions at launch; expand to 200 within 3 months"]]

Minimum threshold to go live: Retrieval recall ≥ [[85%]], faithfulness ≥ [[90%]], citation accuracy = 100%.

Ongoing monitoring

• Run the full eval set after every corpus update.
• Track faithfulness score weekly - a drop of more than [[5 percentage points]] triggers a prompt review.
• Log user thumbs-down clicks and review them weekly; any thumbs-down cluster is a retrieval gap.
• Review 10 random live answers per week against the source documents.

---

Section 10 - Refresh and Re-Index Cadence

A knowledge assistant is only as good as its index. Define when and how the index is updated.

Re-indexing triggers

Partial update method (preferred): Use the document `source_id` metadata field to delete and replace only the chunks from changed documents - do not re-index the entire corpus unless the chunking strategy or embedding model changes.

Full re-index triggers (do these rarely): Switching embedding models, changing chunk size, adding a new metadata field across all documents.

Re-index schedule

Re-index monitoring: Set an alert if the document count in the vector store drops by more than [[5%]] in a single run - this signals an accidental bulk delete.

---

Section 11 - No-Code vs. Build Options

Match the build path to the user's technical comfort and corpus size.

Your technical comfort: [[Non-technical / Somewhat technical / Developer]]

Your recommended path: [[Select from table above based on comfort + corpus size]]

Why: [[One sentence explaining the match - e.g., "Developer comfort + 10,000 docs + compliance requirement → Full build with LlamaIndex + Qdrant self-hosted"]]

---

Section 12 - Cost Model

Per-query cost breakdown

Estimated queries per month: [[e.g., "500 / 5,000 / 50,000"]]

Estimated total monthly cost at that volume:

Your estimated monthly cost: [[Fill in based on your volume + path]]

Cost-control levers

1. Use a smaller model (GPT-5 Instant, Gemini Flash) for routine questions; escalate to a larger model only when confidence is low.

2. Cache answers for high-frequency identical queries - the top 20% of queries are often the same 10 questions.

3. Set a hard spend cap in your LLM provider's dashboard before going live.

4. Reduce top-k - fewer chunks passed to the LLM lowers token cost per query with minimal quality loss if your reranker is tuned.

5. Batch re-index jobs during off-peak hours to avoid paying premium API rates.

---

Section 13 - Launch Checklist

Before the assistant goes live, verify every layer.

Pre-launch verification

1. Chunking audit: Manually inspect 20 random chunks. Confirm each is self-contained and includes the context header and metadata.

2. Retrieval spot-check: Run 10 known test queries. Confirm the correct source document appears in the top-5 retrieved chunks for every query.

3. Reranker validation: Compare ranked vs. unranked top-5 for 10 queries. Confirm reranking moves the correct chunk to position 1 or 2 in at least 8 of 10 cases.

4. Hallucination test: Run 5 queries where the answer is NOT in the corpus. Confirm the assistant says "I don't have that information" rather than inventing an answer.

5. Citation check: Verify every citation in 10 answers is a real, clickable source that contains the stated fact.

6. Access control test: Log in as a restricted-access user and confirm confidential-tier documents do not appear in answers.

7. PII audit: Run a query that should surface a document containing PII. Confirm PII is not exposed to unauthorized users.

8. Eval baseline: Run the full evaluation test set. Confirm retrieval recall ≥ [[85%]] and faithfulness ≥ [[90%]] before go-live.

9. Cost-per-query check: Run 50 test queries; calculate actual average cost per query; compare to estimate.

10. Fallback behavior: Confirm the system degrades gracefully if the vector store is unreachable - returns a clear error message, does not hallucinate.

Sign-off

---

Worked Example

*A concrete fill-in showing what a completed plan looks like for a common scenario.*

Scenario: A 50-person SaaS company wants an internal assistant that answers employee questions about HR policies, the employee handbook, and benefits documents - without the HR team fielding the same 20 questions every week.

Use case definition:
- Audience: All employees (internal, login required)
- Primary use case: Answer HR policy questions using the employee handbook, benefits guide, and PTO policy
- Query style: Conversational questions - "How many sick days do I get?" / "What's the parental leave policy?"
- Acceptable format: 2–3 sentence answer + citation to source document

Document inventory:

Chunking: 600 tokens, 100-token overlap, recursive text splitting. Context header prepended to every chunk.

Embedding + vector store: `text-embedding-3-small` (cost-efficient, English-only) + Pinecone Serverless (managed, zero infra).

Retrieval: Hybrid (semantic + BM25 in Pinecone). Retrieve top-20, rerank with Cohere `rerank-english-v3.0`, pass top-5 to LLM.

Generation model: GPT-5 Instant - sufficient quality for HR Q&A, cost-efficient at ~2,000 queries/month.

Access control: Login required (Okta SSO). All documents are `access_tier = "internal"` - no row-level splitting needed for this corpus.

Evaluation: RAGAS baseline at launch - retrieval recall 91%, faithfulness 94%. Thumbs-down button feeding a weekly Notion review log.

Re-index schedule: PTO Policy re-indexed on every Google Doc change event (Zapier watch). Handbook and Benefits Guide re-indexed annually after new versions published. FAQ CSV re-indexed nightly.

Interface: Flowise chat widget embedded in the company intranet. Source links open the original PDF page.

Estimated monthly cost at 2,000 queries/month: Pinecone Serverless ($0/mo on free tier at this volume) + OpenAI API (~$4/mo) + Cohere reranker (~$1/mo) + Flowise self-hosted ($0) = ~$5/month total.

---

*This template is a planning guide, not a guarantee of specific tool behavior or pricing. Verify API terms, pricing, and data processing agreements directly with each vendor before processing customer data. Pricing figures are approximate as of mid-2025 and subject to change.*