How to Scope a Two-Week AI Pilot (and What It Should Deliver)

Short answer: a two-week AI pilot should deliver one working slice of a real workflow, tested on your own data, plus a written go/no-go decision backed by numbers you agreed on before the pilot started. Not a demo, not a strategy deck. To scope one, pick a single repetitive task, name the person who does it today, collect 50–100 real examples, and write down three pass/fail criteria on Day 1. If you can’t do those four things, the pilot isn’t ready to start.

We run pilots like this for clients, and the pattern below is the one we use. It’s written so you can run it yourself, with or without us.

What “done” looks like after two weeks

At the end of a well-scoped pilot you should be holding five things:

1. A working slice. The AI step runs on real inputs from the real workflow, even if it’s a script or a simple internal page rather than a polished app.
2. A test set. 50–100 real examples with the answer a competent person would give, so you can re-run the test any time the prompt or model changes.
3. A scorecard. Accuracy, time saved per item and the count of harmful mistakes, measured against the targets you set on Day 1.
4. A cost estimate. What one item costs in API usage at the pilot’s volume, and what it would cost at full volume.
5. A decision memo. One page: go, change course, or stop, and why. “Stop” is a perfectly good outcome; it’s two weeks spent instead of six months.

If a vendor’s pilot proposal can’t name all five, ask why. We’ve written before about why most agent pilots stall before production; the common thread is that nobody defined “done” up front.

Step 1: Pick the right task (Day 1)

Good pilot tasks share a shape. Run every candidate through this checklist and pick the one that ticks the most boxes:

• Repetitive: it happens many times a week, in roughly the same way.
• Text-heavy: the inputs are emails, forms, documents, tickets or chat messages.
• Checkable: a person can tell in under a minute whether the output is right.
• Owned: one named person does this job today and will review the AI’s work.
• Low blast radius: a wrong answer gets caught before it reaches a customer or a bank account.
• Examples exist: you can pull 50–100 past cases without a data project.

Typical winners: drafting replies to common support questions, pulling fields out of invoices or forms, tagging incoming requests, summarising long threads for handover. Typical losers: anything that needs data you don’t have yet, anything where “correct” is a matter of taste, and anything that moves money without a human in between.

Step 2: Write the scope on one page (Day 1)

Copy this template into a shared doc and fill it in with the task owner before any code is written:

PILOT SCOPE
Task:          (one sentence, e.g. "Draft replies to delivery-status emails")
Owner:         (the person who does this today and will review outputs)
Inputs:        (where the data comes from, and the format)
Output:        (what the AI produces: a draft, a tag, a filled form)
Human step:    (who approves before anything is sent or saved)
Test set:      (how many real examples, and who writes the correct answers)
Success:       accuracy >= ___ %, median review time <= ___ min/item,
               harmful mistakes = ___
Out of scope:  (integrations, edge cases and features we are NOT doing)
Data rules:    (what may be sent to an AI provider, what must be removed)

The “Out of scope” line matters more than it looks. Two weeks disappear fast once someone asks for a CRM integration on Day 4. Park those requests in a list for the production phase.

The “Human step” line is where most of the trust comes from. Our post on designing approval gates goes deeper on where to put that step and what the reviewer should see.

Step 3: Build the test set before the prompt (Days 2–3)

Pull 50–100 real past cases. Include the awkward ones on purpose: the half-filled forms, the angry emails, the scans at an angle. For each case, the task owner writes down what a good answer looks like. Remove personal data you don’t need, following your own data rules.

Doing this before you write a single prompt keeps everyone honest. Otherwise it’s very tempting to tune the prompt until the demo examples look good, then discover it falls over on the real mix.

Step 4: Build the thinnest working slice (Days 3–8)

Keep the build as small as possible: read an input, call the model, show the output to the reviewer, record their verdict. A script that reads a folder and writes a spreadsheet is fine. If the task is answering questions from your documents, our FAQ assistant tutorial is a working starting point you can adapt in an afternoon.

Practical rules for this stage:

• Start with a mid-sized model such as claude-sonnet-5-5. Try a smaller, cheaper one like claude-haiku-5-5 once you have a score to compare against.
• Ask for structured output (fields, tags, a confidence note) so results can be checked automatically where possible.
• Log every input, output, model id and prompt version. You’ll need them for the scorecard and for debugging.
• Change one thing at a time and re-run the whole test set after each change.

Step 5: Run the reviewer pass and score it (Days 9–11)

The task owner reviews every test-set output and marks it correct, wrong or harmful (wrong in a way that would have caused real trouble), and notes how many minutes the review took. Save that as a CSV:

case_id,verdict,review_minutes
1,correct,2.0
2,correct,1.5
3,wrong,4.0
4,correct,2.5
5,correct,1.0

Then score it against the targets from your one-page scope. This small script does exactly that, using only the Python standard library:

import csv, sys

# Fill these in on Day 1 of the pilot, before anyone sees results.
CRITERIA = {
    "min_accuracy": 0.90,        # share of test cases the reviewer marked correct
    "max_minutes_per_item": 3.0, # median human time per item WITH the AI draft
    "max_bad_misses": 0,         # wrong answers marked "harmful" by the reviewer
}

def median(xs):
    xs = sorted(xs)
    mid = len(xs) // 2
    return xs[mid] if len(xs) % 2 else (xs[mid - 1] + xs[mid]) / 2

def score(path):
    with open(path, newline="") as f:
        rows = list(csv.DictReader(f))
    n = len(rows)
    correct = sum(r["verdict"] == "correct" for r in rows)
    harmful = sum(r["verdict"] == "harmful" for r in rows)
    minutes = median([float(r["review_minutes"]) for r in rows])
    results = {
        "accuracy": (correct / n, correct / n >= CRITERIA["min_accuracy"]),
        "median_minutes": (minutes, minutes <= CRITERIA["max_minutes_per_item"]),
        "harmful_misses": (harmful, harmful <= CRITERIA["max_bad_misses"]),
    }
    print(f"{n} test cases")
    for name, (value, ok) in results.items():
        shown = f"{value:.0%}" if name == "accuracy" else value
        print(f"  {name:15} {shown!s:>6}  {'PASS' if ok else 'FAIL'}")
    go = all(ok for _, ok in results.values())
    print("Decision:", "GO - plan the production build" if go else "NO-GO - fix or stop")
    return go

if __name__ == "__main__":
    score(sys.argv[1] if len(sys.argv) > 1 else "pilot_results.csv")

Run python3 scorecard.py pilot_results.csv. On the five sample rows above it prints:

5 test cases
  accuracy           80%  FAIL
  median_minutes     2.0  PASS
  harmful_misses       0  PASS
Decision: NO-GO - fix or stop

To make the time number meaningful, have the owner also time a handful of cases done the old way, without the AI draft. “Two minutes per item” only means something next to “what it took before”.

Step 6: Cost it and write the decision (Days 12–14)

From your logs, take the average input and output tokens per item, multiply by your provider’s current per-token prices, then by your monthly volume. Add a line for the people-time to review, and a line for upkeep: prompts, model upgrades and the test set all need maintaining. We break that last part down in the maintenance bill nobody quotes for AI features.

Then write the one-page memo:

• Go: all criteria passed. List what production needs: integrations from the “out of scope” list, access controls, monitoring, and who owns it.
• Change course: close on some criteria, with a clear reason for the misses (for example, scanned PDFs fail but typed ones pass). Narrow the scope and run one more week.
• Stop: far off, or the review takes as long as doing the job by hand. Record why so nobody re-runs the same pilot next year.

Common scoping mistakes

• Two tasks in one pilot. You end up with two half-tested slices and no clear answer on either.
• No named reviewer. Without the person who does the job today, nobody can say whether the output is right.
• Criteria set after the results. Targets picked after seeing the score will always be met. Write them on Day 1.
• Demo data instead of real data. Clean examples flatter every model. The awkward 20% is where the pilot earns its keep.
• Integration work in week one. Connect to live systems after the go decision, not before.

Want a hand scoping yours?

Jumpstart Creatives runs short, fixed-scope AI pilots for teams that want a clear answer before they commit a budget: we help pick the task, build the test set and the working slice, and hand over the scorecard and decision memo whatever the outcome. See what we offer on our services page, or get in touch and tell us which task you’d pilot first.