RAG for customer support: how to make AI replies actually accurate

Ask a plain LLM “how do I reset my password?” and you get a plausible answer that is probably wrong: a different button, a different flow, a different URL. RAG (retrieval-augmented generation) fixes this by putting the right passages from your own content into the prompt before the model answers.

The pipeline at a glance

Three stages: ingest, retrieve, generate.

1. Ingest

• Upload PDF, DOCX, TXT, Markdown, CSV or JSON files, crawl your website, or add Q&A pairs under Training.
• The worker extracts the text and splits it into passages of about 2,000 characters (roughly 500 tokens), with a 200-character overlap. Each Q&A pair stays one passage.
• Each passage is embedded with text-embedding-3-small (1,536 dimensions).
• Passages are stored in PostgreSQL with the pgvector extension and an HNSW index for cosine distance.

2. Retrieve

When a visitor's message needs facts, we embed their recent messages and run a cosine-distance search over the workspace's passages. Greetings and small talk skip this step. A simplified version of the query:

-- Simplified. The real query also checks that the document is
-- ready, enabled, in date and visible on the visitor's platform.
SELECT c.id, c.content,
       1 - (c.embedding <=> $1::vector) AS similarity
FROM knowledge_chunks c
WHERE c.tenant_id = $2
ORDER BY c.embedding <=> $1::vector
LIMIT $3;  -- 3 × top-K candidates, re-ranked afterwards

We fetch three times as many candidates as we need, re-rank them, drop anything under the similarity threshold (about 0.3), keep at most two passages per document, and pass the top five (by default) to the model.

3. Generate

The passages are appended to the system prompt inside a fenced reference block. The model is told to treat that block as data, not instructions, to use only the references that answer the question, and not to invent product facts that aren't there.

If a visitor asks about a product fact and nothing relevant is found, the model isn't called at all: the visitor is told the information isn't available, and the chat goes to a person unless automatic hand-off is turned off.

Keeping the cost down

Three things keep a large knowledge base affordable:

• Embedding cache: the same search text gets the same embedding. The key is the model name plus a SHA-1 hash of the text, kept in Redis for 24 hours.
• Prompt caching: for Claude models the request asks Anthropic to cache the prompt, and OpenAI caches long prompts automatically. The reference block changes with every reply, so mostly the fixed part of the prompt benefits.
• Threshold-based skipping: if no passage passes the threshold, nothing is added to the prompt.

When RAG isn't enough

RAG handles “how do I…?” questions well. It can't answer “where is my order?”, because that data isn't in your documents.

For that, AskAIs has Custom tools: you describe an HTTP endpoint in your own system, such as orders, bookings or stock, and the AI can call it while it writes the reply. General questions are answered from your knowledge, personal ones from your systems, and the rest goes to your team.