How to Write an Evaluation Set for an AI Feature in One Afternoon

Short answer: an evaluation set is a spreadsheet of real inputs, each paired with a simple pass/fail check, that you run every time you change a prompt, model or retrieval setting. You can build a useful first version in one afternoon: pull 30–60 real questions from your inbox or chat logs, tag each by type, write one mechanical check per row (contains, regex, valid JSON, “says I don’t know”), and run them with a 70-line script. It won’t be perfect. It will stop you shipping a change that quietly breaks something that used to work.

Why an AI feature needs an eval set before it needs a better prompt

Language model features fail differently from normal code. A prompt tweak that fixes one answer can break three others, and nobody notices until a customer does. Without a fixed set of test inputs, every change gets judged by whoever tried “a few questions” that morning.

An eval set turns “it feels better” into “it passed 52 of 55, and here are the three it failed.” That is the number you put in a pilot go/no-go decision (we covered that in how to scope a two-week AI pilot), and it is what lets you swap to a cheaper model later without guessing.

The afternoon plan

1. Hour 1: collect real inputs. 30–60 is enough to start.
2. Hour 2: tag and write checks. One tag and one check per row.
3. Hour 3: run the script, read every failure. Fix the checks that are wrong before you fix the feature.
4. Last 30 minutes: wire it into your routine. A threshold, a results file, and a rule that nobody changes the prompt without running it.

Step 1: collect inputs from reality, not imagination

The most common mistake is writing test questions yourself. You will write the questions you already know the system handles. Instead, pull them from where real users already ask:

• Support inbox and chat transcripts from the last few months
• Search logs on your site or help centre
• Questions your sales or front-desk team answers repeatedly
• Past failures: anything a customer complained the bot got wrong

Copy them as written, typos and all. Strip names, phone numbers and account details first. Then deliberately add a handful of cases the feature should refuse: off-topic questions, requests for things it has no data on, and anything that would be harmful to answer confidently. A good starting mix is roughly 70% common questions, 20% edge cases, 10% things it must decline. That ratio is a rule of thumb, not a law; adjust it to what you see in your own logs.

Step 2: tag each row and pick one mechanical check

Put everything in a CSV with five columns: id, tag, input, check, expected. Tags let you see where the feature is weak (refunds fine, pricing broken) instead of one blended score.

Use the simplest check that would catch a wrong answer:

• contains: the answer must mention a key fact (“30 days”, “refund form”).
• not_contains: the answer must not say something (a competitor’s name, a discontinued plan).
• regex: a pattern, such as a price format.
• json_keys: output must be valid JSON with certain keys, for features that feed other systems.
• refuses: the answer must admit it doesn’t know.

Mechanical checks miss nuance, and that is fine for day one. They are fast, free to run and never have a bad mood. You can add model-graded or human-graded checks later for the rows where wording really matters.

id,tag,input,check,expected
1,refunds,"Can I get a refund after 30 days?",contains,30 days
2,refunds,"How do I request a refund?",contains,refund form
3,shipping,"Do you ship to Cebu?",contains,Cebu
4,out_of_scope,"What's the weather tomorrow?",refuses,
5,format,"Give me your office hours as JSON",json_keys,open;close
6,pricing,"How much is the Pro plan?",regex,"PHP\s?[\d,]+"

Note the quotes around the regex in row 6. Any field containing a comma has to be quoted, or the CSV reader splits it. We hit exactly that while testing this post.

Step 3: the runner script

This script reads the CSV, sends each input to your feature, applies the check, prints every failure, shows a pass rate per tag, saves the raw outputs to a timestamped file, and exits non-zero if the pass rate drops below 90%. It runs offline against a stand-in function so you can try it immediately; pass --live to call a real model through the Anthropic Python SDK (pip install anthropic, with ANTHROPIC_API_KEY set).

"""Tiny eval runner: python run_evals.py evals.csv [--live]"""
import csv, json, re, sys
from collections import defaultdict
from datetime import datetime

REFUSAL_MARKERS = ("i don't know", "i'm not sure", "can't help with", "outside what i can")

def check(output: str, kind: str, expected: str) -> bool:
    out = output.lower()
    if kind == "contains":
        return expected.lower() in out
    if kind == "not_contains":
        return expected.lower() not in out
    if kind == "regex":
        return re.search(expected, output) is not None
    if kind == "refuses":
        return any(m in out for m in REFUSAL_MARKERS)
    if kind == "json_keys":
        try:
            data = json.loads(output)
        except json.JSONDecodeError:
            return False
        return isinstance(data, dict) and all(k in data for k in expected.split(";"))
    raise ValueError(f"unknown check type: {kind}")

def fake_feature(question: str) -> str:
    """Stand-in so the runner works offline. Replace with your real feature."""
    canned = {
        "refund after 30": "Refunds are only available within 30 days of purchase.",
        "request a refund": "Fill in the refund form on your account page.",
        "ship to cebu": "Yes, we ship nationwide, including Cebu.",
        "office hours": '{"open": "09:00", "close": "18:00"}',
        "pro plan": "The Pro plan is PHP 1,499 per month.",
    }
    for key, answer in canned.items():
        if key in question.lower():
            return answer
    return "I don't know. That's outside what I can answer."

def live_feature(question: str) -> str:
    import anthropic
    client = anthropic.Anthropic()
    msg = client.messages.create(
        model="claude-haiku-5-5", max_tokens=400,
        system="You are our support assistant. If unsure, say \"I don't know\".",
        messages=[{"role": "user", "content": question}],
    )
    return msg.content[0].text

def main(path: str, feature) -> float:
    rows = list(csv.DictReader(open(path, newline="", encoding="utf-8")))
    by_tag = defaultdict(lambda: [0, 0])
    results = []
    for row in rows:
        output = feature(row["input"])
        ok = check(output, row["check"], row["expected"])
        by_tag[row["tag"]][0] += ok
        by_tag[row["tag"]][1] += 1
        results.append({**row, "output": output, "pass": ok})
        if not ok:
            print(f"FAIL #{row['id']} [{row['tag']}] {row['input']!r}\n  got: {output[:200]!r}")
    passed = sum(r["pass"] for r in results)
    print(f"\n{passed}/{len(results)} passed")
    for tag, (p, n) in sorted(by_tag.items()):
        print(f"  {tag:<14} {p}/{n}")
    stamp = datetime.now().strftime("%Y%m%d-%H%M")
    with open(f"results-{stamp}.jsonl", "w", encoding="utf-8") as f:
        for r in results:
            f.write(json.dumps(r, ensure_ascii=False) + "\n")
    return passed / len(results)

if __name__ == "__main__":
    feature = live_feature if "--live" in sys.argv else fake_feature
    score = main(sys.argv[1], feature)
    sys.exit(0 if score >= 0.9 else 1)

Run it with python run_evals.py evals.csv. Output looks like this:

6/6 passed
  format         1/1
  out_of_scope   1/1
  pricing        1/1
  refunds        2/2
  shipping       1/1

Replace fake_feature (or live_feature) with a call into your actual feature: the same function your app uses, with the same system prompt and retrieval. If you built something like our FAQ assistant on your own documents, that means calling its answer() function, not a bare model call. An eval that bypasses your real pipeline tests the wrong thing.

Step 4: read every failure, and suspect the check first

When we changed the stand-in’s pricing answer to “1,499 pesos” instead of “PHP 1,499”, the run reported:

FAIL #6 [pricing] 'How much is the Pro plan?'
  got: 'The Pro plan is 1,499 pesos per month.'

5/6 passed

The answer is correct; the check is too strict. This happens a lot in the first runs. For each failure, decide which of three things is wrong:

1. The check is too narrow. Loosen it (accept “PHP” or “pesos”).
2. The expected answer is outdated. Update it from the source of truth.
3. The feature is wrong. Now you have a real bug, with a reproducible input.

Only the third kind should change your prompt. If you tune the prompt to satisfy a bad check, you make the product worse while the score goes up.

Step 5: make it a habit, not a one-off

• Run it on every change. Prompt edits, model upgrades, new documents in the knowledge base, retrieval settings. The exit code means you can drop it into CI or a pre-deploy script.
• Keep the results files. The JSONL output lets you diff two runs and see exactly which answers changed, not just the score.
• Add a row for every production bug. When a user reports a bad answer, it goes in the CSV before it gets fixed. The set grows from real failures and the same bug can’t come back silently.
• Set the threshold deliberately. 90% is a placeholder. For a feature that drafts replies a human reviews, lower may be fine; for one that sends answers straight to customers, you may want 100% on the “must refuse” tag specifically.
• Re-run on a schedule even when nothing changed. Hosted models get updated and your documents drift. We go into that ongoing cost in the maintenance bill for AI features.

Common mistakes we see

• Only happy-path questions. A set with no refusal cases will happily reward a feature that makes things up.
• One overall score. 92% can hide a tag sitting at 40%. Always look at the per-tag lines.
• Checks that test wording. Matching a full expected sentence fails on every rephrasing. Check facts, not phrasing.
• Letting the set go stale. If nobody adds rows after launch, it stops reflecting what users actually ask.
• Running it on a different pipeline. Same prompt, same retrieval, same model settings as production, or the number means nothing.

What to do next

Block out an afternoon, export last month’s support questions, and get the first 30 rows into a CSV. The script above will do the rest. Once you have a baseline, every future change to the feature becomes a measurable decision instead of a judgement call.

If you’d rather have help setting this up, or you have an AI feature in production with no test set behind it, that’s the kind of work we do. See our services or get in touch and tell us what the feature does; we’ll tell you what we’d test first.