How to Build an FAQ Assistant on Your Own Documents (Step-by-Step Tutorial)

Short answer: you can build a useful FAQ assistant on your own documents in an afternoon. Split your documents into small passages, find the few passages that match each question, and ask a language model to answer only from those passages, saying “I don’t know” when they don’t cover it. That last rule does more for trust than any other part of the build.

This tutorial walks through a complete, working version in about 80 lines of Python. It is deliberately simple: no vector database, no framework, nothing you can’t read in one sitting. It’s the version we’d build first for a client before deciding whether anything heavier is worth paying for.

What you’ll build

• A folder of Markdown or text files (policies, product docs, onboarding notes) becomes a searchable knowledge base.
• A question comes in, the script picks the most relevant passages, and the model answers from them with the source file named.
• If the documents don’t contain the answer, the assistant says so instead of guessing.

What you need

• Python 3.10 or newer
• An Anthropic API key, set as the ANTHROPIC_API_KEY environment variable (any capable model provider works; only the call in step 4 changes)
• Two packages: pip install anthropic rank-bm25
• A folder called docs/ with a handful of .md or .txt files

Step 1: Split documents into passages

Models answer better from a few relevant paragraphs than from a whole manual. Split each file on blank lines and merge short paragraphs until each passage is a readable chunk, a few hundred words at most. Keep the file name with every passage so the answer can cite it.

from pathlib import Path

def load_passages(folder="docs", max_chars=1200):
    passages = []
    for path in sorted(Path(folder).glob("*.[mt][dx]*")):
        buf = ""
        for para in path.read_text(encoding="utf-8").split("\n\n"):
            para = para.strip()
            if not para:
                continue
            if len(buf) + len(para) > max_chars and buf:
                passages.append({"source": path.name, "text": buf})
                buf = ""
            buf = f"{buf}\n\n{para}" if buf else para
        if buf:
            passages.append({"source": path.name, "text": buf})
    return passages

Step 2: Find the relevant passages

For a first version, keyword search is enough, and it’s easier to debug than embeddings: you can see exactly why a passage matched. BM25 is the standard ranking method behind most search boxes, and the rank-bm25 package implements it in one line.

import re
from rank_bm25 import BM25Okapi

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

class Retriever:
    def __init__(self, passages):
        self.passages = passages
        self.bm25 = BM25Okapi([tokenize(p["text"]) for p in passages])

    def top(self, question, k=4):
        scores = self.bm25.get_scores(tokenize(question))
        ranked = sorted(zip(scores, self.passages), key=lambda x: x[0], reverse=True)
        return [p for score, p in ranked[:k] if score > 0]

When people start asking questions in words that never appear in your documents (“leave” versus “time off”), that’s the signal to add embeddings. Until then, keyword search keeps the system cheap and explainable. We compared the two approaches in Retrieval or a bigger context window?

Step 3: Write the instructions that keep it honest

The system prompt is where you decide how the assistant behaves when it isn’t sure. Be explicit:

SYSTEM = """You answer questions for employees using ONLY the passages provided.
Rules:
- If the passages do not contain the answer, reply exactly:
  "I couldn't find that in our documents." and suggest who to ask.
- Never use outside knowledge, even if you are confident.
- Quote numbers, dates and amounts exactly as written.
- End with the source file names you used, like: (Source: leave-policy.md)"""

Step 4: Ask the model

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY

def answer(question, retriever):
    hits = retriever.top(question)
    if not hits:
        return "I couldn't find that in our documents."
    context = "\n\n".join(f"[{p['source']}]\n{p['text']}" for p in hits)
    msg = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=600,
        system=SYSTEM,
        messages=[{"role": "user",
                   "content": f"Passages:\n{context}\n\nQuestion: {question}"}],
    )
    return msg.content[0].text

Notice the early return: if search finds nothing, we never call the model at all. That saves money and removes the most common source of made-up answers.

Step 5: Put it together

if __name__ == "__main__":
    retriever = Retriever(load_passages("docs"))
    print(f"Loaded {len(retriever.passages)} passages. Ask away (Ctrl+C to quit).")
    while True:
        q = input("\n> ").strip()
        if q:
            print(answer(q, retriever))

Run it with python faq.py and ask something your documents cover, then something they don’t. The second test matters more than the first.

Test it before anyone else does

Before you share it, write down 20 real questions people have actually asked, with the answer you expect. Include five that the documents can’t answer. Run all 20 after every change. You’re checking three things:

1. Right answer, right source. Does it cite the file the answer really came from?
2. Honest refusals. Does it say “I couldn’t find that” for the five unanswerable questions, rather than improvising?
3. Exact figures. Are amounts, dates and deadlines copied correctly?

This small test set is the most valuable artifact of the whole project. It tells you when a change to the documents, the prompt or the model has made things worse.

Where this version stops

This is a solid internal tool for a team. Before putting it in front of customers, you’d typically add:

• Embeddings or hybrid search once vocabulary mismatches show up in your test set.
• Access control, so people only retrieve documents they’re allowed to read.
• Logging and review of real questions, which is how you find gaps in the documents themselves.
• A human hand-off for anything involving money, refunds or account changes. We wrote about this in Designing approval gates.
• A plan for upkeep. Documents change, and the assistant is only as current as its sources. See The maintenance bill for AI features.

Want this built for your team?

Jumpstart Creatives builds AI assistants, agents and the software around them for businesses in the Philippines and abroad, from a two-week pilot like this one to production systems with access control, evaluation and monitoring. If you’d like help taking this from a script to something your customers can use, tell us about the project or see what we do.