All insights
AI

Before You Trust an AI Automation, Test It on 20 Real Jobs

19 Nov 2026 · 5 min read

Picture a plumbing business that connects an AI assistant to its inbox. In the demo it books three jobs perfectly. Then, in its first live week, it books a customer in on a public holiday, reads "ASAP" as next Tuesday, and says a quote's been sent while the accounting system still shows a draft. Every one of those was ordinary work. A short test on the business's own jobs would've caught each one before a customer did.

The test is simple. Take 20 jobs you'd normally handle, write down what should happen for each, and check the result in the system that owns it. While you test, a person approves anything that touches customers, money, records, bookings or live systems.

Build a test set from work you already know

You'll want three kinds of job:

  • 10 normal jobs. These show whether it copes with your everyday volume.
  • 5 awkward jobs. Missing details, odd wording, duplicate requests, changed instructions, a blurry scan, a system that's down.
  • 5 jobs it should hand to a person. A refund above your limit, a request for sensitive information, a change of bank details, or a decision that's above its pay grade.

Swap real customer details for made-up ones in your test tools, and use real customer material only where your privacy review says it's OK.

Write the expected outcome first

Judge the workflow on what actually lands in your systems. For each case, write what must happen, what must stay untouched, and what proves it's done. A booking test passes when the right booking is in the calendar and the confirmation went to the right address. An invoice test passes when the amounts, customer details, GST and status are right in your accounting system. That way only finished work counts as a pass, whatever the "all done!" message says.

Use a scorecard your team can repeat

Make one row per test with these columns:

  • Case and the real-world input.
  • Expected final outcome.
  • Checks or approvals it needs.
  • Actions outside its job.
  • What actually happened in the connected system.
  • Time a person spent reviewing.
  • Pass, fail or look closer.
  • Failure note and who owns the fix.

Keep your failed examples. They're your test set for the next prompt, model, integration or process change.

Set your launch rules before you see results

A good pass rate can still hide a costly mistake, so set separate rules up front: zero unauthorised actions, every handoff reaches a person, business facts come from an approved source, and every high-stakes case passes. Then pick a quality target for normal jobs that you're happy to sign off. If the workflow saves you two minutes but adds five minutes of checking, keep it supervised for now. AI answers can vary from run to run, so run your important cases more than once.

Test the whole workflow

Problems often sit around the AI rather than in it: an old price list, permissions that are too broad, a broken connection, nobody owning the task, or no alert when something stops. So test with the same tools, permissions, notifications and review steps you'll use live. Switch off a connected system during one test and check the work stops safely and tells someone. Then write down who can pause the automation, how staff report a bad result, and where you can see what it did.

Re-run the tests when something important changes

Run the same set again whenever you change the model, the instructions, the connected data, the permissions or the process itself. Add a new case every time a real job trips it up. Over time, your one-off launch check becomes a habit, and it gives you evidence for the next call: expand it, keep it supervised, narrow its job, or switch it off.

Takeaway: start with 20 real jobs and a written scorecard. Go live when the final outcomes are right, unauthorised actions stay at zero, handoffs reach a person, and checking takes less time than doing the job yourself.

Want help running this test on your own workflow? Our Grow plans cover automation of the manual work eating your week and integrations between the systems you already run, with a prioritised list of what to do next reviewed with you monthly. To talk through your 20 cases first, book a discovery call.

Source note: This original Maud Digital guide was informed by the OAIC's guidance on privacy and the use of commercially available AI products, which calls for testing before deployment, ongoing monitoring, and human oversight of AI outputs, and by the Australian Government's Guidance for AI Adoption (Oct 2025). It is general operational guidance; seek legal, privacy, and cybersecurity advice for your situation.

General information to help you decide, based on public sources at the date shown. For legal, tax or compliance questions about your business, speak with a qualified adviser.

let's talk

Want technology decisions like these made for you?

book a discovery call