Build a RAG Knowledge-Base AI Agent in n8n

What RAG is, and when to use it

A language model only knows what was in its training data and what you put in the prompt. Ask it about your refund policy, your API changelog, or last quarter’s internal runbook and it will either decline or, worse, confidently make something up. Retrieval-Augmented Generation closes that gap by fetching relevant passages from a knowledge base you control and placing them in the prompt before the model answers. The model still writes the sentence, but the facts come from your documents, and you can show exactly which ones.

RAG is one of three ways to give a model knowledge it did not ship with, and they are not interchangeable. Knowing which to reach for saves weeks of wasted effort.

RAG vs fine-tuning vs long-context

Fine-tuning changes the model’s weights by training on examples. It is the right tool for teaching a consistent format, tone, or skill (always reply as JSON, always write in our brand voice), but it is a poor and expensive way to teach facts. Facts change; retraining every time your pricing page updates is absurd, and the model still cannot cite where an answer came from.

Long-context prompting means stuffing whole documents into the prompt window. Modern models accept very large contexts, so for a handful of documents this is simpler than RAG and worth doing first. It breaks down on volume: cost and latency scale with every token you send, and retrieval accuracy actually degrades when the relevant fact is buried in a mountain of irrelevant text. Sending 200 pages to answer one question is slow, costly, and often less accurate than sending the three paragraphs that matter.

RAG is the answer when your knowledge base is large, changes often, and you need citations. You index once, retrieve a few relevant chunks per query, and pay only for those tokens. Updating knowledge means re-indexing a document, not retraining a model. For most “answer questions about our stuff” use cases, RAG is the default, and keeping the per-query token footprint small is also the simplest lever on LLM token cost.

Chunking: the decision that makes or breaks recall

You do not embed whole documents. You split them into chunks, embed each chunk, and retrieve at the chunk level. Get this wrong and nothing downstream can save you: chunks that are too large dilute the signal and waste tokens, chunks that are too small lose the context that makes a passage meaningful, and chunks that ignore document structure split a table or a numbered procedure straight down the middle.

Three rules cover most cases. First, size by meaning, not by character count alone. A good default for prose is roughly 600 to 1,000 tokens per chunk. Second, add overlap of around 10 to 20 percent so a sentence cut at a chunk boundary still appears whole in the neighbouring chunk. Third, split on structure where you can: headings, paragraphs, then sentences, in that order of preference. n8n’s Recursive Character Text Splitter does exactly this when you give it an ordered list of separators.

# n8n: Recursive Character Text Splitter (under Default Data Loader)
chunk_size: 800            # tokens; 600-1000 suits most prose
chunk_overlap: 120         # ~15% of chunk_size, preserves cross-cut context
separators: ["\n\n", "\n", ". ", " "]   # try paragraphs, then lines, then sentences
keep_metadata:
  - source_url
  - title
  - section
  - updated_at            # enables freshness filters and targeted re-indexing

Attaching metadata at chunk time is not optional polish. The source_url and title are what let the model cite a chunk; section powers metadata filtering at retrieval; updated_at is what lets you re-index only what changed. Structured inputs such as PDFs and scanned forms usually need an extraction step first, which is its own discipline covered in document processing automation.

Choosing an embedding model

An embedding model turns a chunk of text into a vector, a list of numbers that captures its meaning, so that passages about the same topic land near each other in space. Retrieval is then just “find the nearest vectors to the question’s vector.” Three practical points matter more than the leaderboard rankings.

Pick one model and stay on it. If you re-embed your corpus with a different model, you must re-embed everything, including future queries, because vectors from two models are not comparable. Switching embedding models is a full re-index, not a config tweak.

Know your options. OpenAI’s text-embedding-3-small and -large are the common default and are wired into n8n directly. Cohere and Google offer competitive hosted models. For open-source and self-hosted setups, BGE, E5, and nomic-embed run well through Hugging Face or a local Ollama instance, which keeps your documents on your own infrastructure. One accuracy note worth stating plainly: Anthropic does not provide a first-party embeddings endpoint and points users to Voyage AI for embeddings. You can absolutely use Claude as the generation model while embedding with OpenAI, Voyage, Cohere, or an open model; the two roles are independent.

Dimensions and language matter. Higher-dimensional vectors capture more nuance but cost more to store and search. If your content is multilingual or domain-heavy (legal, medical, code), test a couple of models on your own questions rather than trusting a generic benchmark.

Grounding, citations, and refusing to guess

Retrieval gets the right context in front of the model. Grounding is what makes the model actually use it and admit when it cannot. Without an explicit instruction, a model will happily blend retrieved facts with its own training data, which is exactly the hallucination RAG was supposed to prevent. The fix lives in the system prompt.

You are a support assistant. Answer ONLY from the provided context.

Rules:
1. Use only facts found in the context below. Do not use outside knowledge.
2. After each claim, cite the source using the title and source_url from
   that chunk's metadata, e.g. [Billing FAQ](https://...).
3. If the context does not contain the answer, reply exactly:
   "I don't have that in the knowledge base yet. Want me to open a ticket?"
4. Never invent prices, dates, or policy. Quote the document instead.

<context>
{{ retrieved_chunks_with_metadata }}
</context>

Question: {{ user_question }}

Three behaviours are doing the work here. Source-only answering stops the model leaning on training data. Inline citations let a human verify any claim in one click and make the system auditable. An explicit refusal path means “I don’t know” is a valid, designed outcome rather than a confident fabrication, which is the single most trust-building thing a knowledge agent can do.

One security caveat that teams forget: retrieved chunks are untrusted text. If your knowledge base ingests anything users or third parties can write into, a malicious document can carry instructions that try to hijack the agent. Treat retrieved content as data, not commands, and harden the workflow accordingly, the patterns in n8n prompt injection hardening apply directly to the context you splice into every prompt.

The build runbook in n8n

n8n’s AI nodes are LangChain under the hood, arranged as a main node with sub-nodes you attach beneath it. The two pipelines map onto two small workflows. Here is the order that gets you from empty canvas to a working, cited agent.

1. Load your sources. Use HTTP Request, Google Drive, Notion, or a webhook to pull documents in, then feed them to a Default Data Loader. This is your ingest workflow’s front door.
2. Split into chunks. Attach a Recursive Character Text Splitter sub-node to the loader with the size, overlap, and separators from the chunking section above.
3. Generate embeddings. Attach an Embeddings sub-node (OpenAI, Cohere, Hugging Face, or Ollama). Note which model and dimensions you chose; you are committing to it.
4. Insert into the vector store. Add a Vector Store node (Postgres PGVector, Qdrant, Pinecone, or Supabase) in Insert mode, with the loader and embeddings wired beneath it. Run this workflow once to build the index.
5. Start the query workflow. Add a chat trigger or webhook, then an AI Agent node, or a Question and Answer Chain for a simpler request-response shape.
6. Attach retrieval. Give the agent a Vector Store node configured as a retrieval tool, pointed at the same store and the same embeddings model used at insert time. The agent now searches your knowledge base on demand.
7. Add the grounding prompt. Paste the system prompt from above into the agent or chain so answers stay source-only, carry citations, and refuse gracefully.
8. Tune retrieval. Set top-k, add metadata filters so queries stay scoped, and turn on hybrid or re-ranking if precision needs it.
9. Schedule re-indexing. Add a Schedule or webhook trigger that re-runs the ingest workflow when documents change, upserting by an id so you replace chunks rather than duplicating them.
10. Evaluate before you ship. Run a fixed set of real questions through the query workflow and score the answers, so you can prove a change helped instead of guessing.

That is the whole system. The architecture choices, which agent pattern, how much to let the model decide versus script, are worth getting right early; our notes on AI agent development cover the trade-offs in depth.

Worked example: a SaaS support knowledge base

Make it concrete. Imagine a SaaS company with a help centre of about 400 articles, a billing FAQ, and an API changelog, all of which change weekly. Support agents waste hours answering the same questions, and customers get inconsistent replies. The goal: a chat agent that answers from the official docs and links the article it used.

Ingest. A nightly workflow pulls every published help-centre article via the CMS API, runs them through the Default Data Loader and the Recursive Character Text Splitter at 800-token chunks with 120 overlap, embeds them with text-embedding-3-small, and upserts into Postgres with pgvector. Each chunk carries title, source_url, product, and updated_at metadata.

A query in flight. A customer asks, “Can I downgrade my plan mid-cycle and get a refund?” The query workflow embeds that question with the same model, retrieves the top 5 chunks filtered to product = "billing", and the nearest matches are two paragraphs from the billing FAQ. The grounding prompt feeds only those chunks to the model, which replies: “You can downgrade at any time; the change and any prorated credit take effect on your next billing date,” followed by a link to the Billing FAQ article it cited.

The refusal that builds trust. The next customer asks about a feature the company has not launched. Retrieval returns nothing relevant, so the model follows its instruction and replies, “I don’t have that in the knowledge base yet. Want me to open a ticket?” instead of inventing a release date. That single behaviour, a clean “I don’t know,” is what makes support teams willing to put the agent in front of real customers.

Keeping it fresh and proving it works

A RAG system is only as good as its index, and an index quietly rots. Two disciplines keep it honest.

Re-indexing on update

Decide what triggers a refresh: a schedule (nightly is plenty for most docs), a webhook from your CMS on publish, or a manual run. The key detail is to upsert by a stable id so editing an article replaces its chunks instead of stacking a second copy beside the old one. The updated_at metadata lets you re-embed only what changed rather than rebuilding the whole corpus every night, which keeps embedding cost flat as the knowledge base grows.

Evaluating retrieval and answers separately

When an answer is wrong, you need to know whether retrieval fetched the wrong chunks or the model misused good ones, because the fixes are different. Build a small evaluation set of real questions with known-good answers and known source documents, then measure two things. Retrieval quality: did the correct chunk appear in the top-k? If not, the problem is chunking, embeddings, or filters. Answer quality: given the right chunks, was the answer faithful and did it cite correctly? If not, the problem is the grounding prompt or the model. Run this set before and after every change so “it feels better” becomes a number you can defend.

What other experts say

Ship-ready checklist before you go live

• Lock one embedding model. Record the exact model and dimension count, and wire the same node into both the insert workflow and the query workflow so vectors stay comparable.
• Stamp metadata at chunk time. Confirm every chunk carries source_url, title, section, and updated_at before it reaches the vector store, since you cannot add these after insert without re-indexing.
• Upsert by a stable id. Set the vector store node to replace chunks for a document id rather than append, then re-run the ingest twice on one doc and verify the row count does not grow.
• Put a refusal path in the system prompt. Ask a question you know is not in the corpus and confirm the agent returns the fallback message instead of inventing an answer.
• Treat retrieved text as untrusted. Feed a document containing a fake instruction and check the agent ignores it, so an ingested page cannot hijack the prompt.
• Set top-k deliberately. Start at 4 to 6, then watch token usage and answer quality on real questions before raising it.
• Build a fixed evaluation set. Collect 15 to 30 real questions with known-good answers and source documents, and run it before and after every retrieval change.
• Schedule re-indexing. Add the trigger that refreshes the index on a schedule or CMS publish, and confirm an edited document’s answer updates after a run.

RAG knowledge agent readiness card