Building reliable AI workflows with human review
The normal case makes a convincing demo: a clear enquiry arrives, the right information is found and a reply is prepared. Everyday work also includes missing details, expired documents, duplicate messages and systems that temporarily stop responding.
A reliable AI workflow needs a designed response to those cases. This guide uses a customer-enquiry workflow to explain the checks, review rules and operational habits worth agreeing before launch. The examples are illustrative, not customer performance claims.
Define completion before automating
For an enquiry, “the model produced a reply” is an intermediate event. Completion might mean a person approved the reply, the customer received it and the customer record contains the result.
If the message is sent but the record update fails, the system should show partial completion. Retrying the entire workflow could send the customer a duplicate message. Separate the steps that can be retried from actions whose previous outcome must be checked first.
Write down the states a colleague needs to recognise: received, awaiting information, ready for review, approved, completed and needs attention. Use names that reflect your actual process. Every state that can stall needs an owner and a next action.
Decide what can happen automatically
Choose permissions action by action. A business may allow an internal classification automatically while requiring approval for an outgoing reply or a change to customer terms.
| Situation | Example handling rule |
|---|---|
| The enquiry matches a known service and current source | Prepare a draft for review |
| The customer identity is ambiguous | Ask for clarification or route to the account owner |
| Two sources disagree about a price | Hold the draft and show the conflict |
| The customer requests an exception | Route to the person authorised to decide |
| A connected tool is unavailable | Keep the task pending and alert the owner after the agreed retry limit |
| A message may already have been sent | Check the delivery record before retrying |
These are example rules, not defaults for every company. The point is to make each boundary visible and enforceable. Do not rely only on a prompt telling the model to be careful.
Give reviewers the evidence they need
A review screen should contain the proposed action, the source material used, the customer or record it applies to, and any missing or conflicting details. Show what will happen when the reviewer approves.
If approval covers a reply and a CRM update, display both. If the underlying quotation changes while the draft waits, re-check it before executing the approved action. Otherwise, a person can approve a correct-looking screen while the system acts on outdated context.
Set a route for work that sits unreviewed. It might stay in the original inbox, appear in a shared queue or be assigned to a backup owner. Choose a place the team actually watches.
Test the awkward cases deliberately
Create a test set from representative examples, with personal information removed or access controlled as appropriate. Define the expected behaviour for each case before inspecting the output. Keep important failure cases alongside the normal ones.
OpenAI's evaluation guidance recommends task-specific evaluations, representative data and continued testing as an application changes. Those principles are useful beyond one model provider. Read the evaluation guidance.
For an enquiry workflow, include:
- A straightforward request with a current, relevant source.
- A request whose answer is absent from the approved material.
- An old document that conflicts with a current one.
- Two customers with similar names.
- An attachment missing the information needed to proceed.
- The same event delivered twice.
- A tool failure after one action has already succeeded.
- Text in an incoming document that tries to tell the system to ignore its rules.
The last case matters because customer messages and retrieved documents are inputs, not instructions that should grant the system new permissions. Keep allowed actions and access checks in the application. Treat model output as something to validate, especially before using it to make a change in another tool.
Measure quality by what the workflow does
Agree checks that a reviewer can apply consistently. For example: correct customer, supported service details, no invented availability, correct exception routing and no repeated external action.
Track the review effort as well. A draft that is technically accurate but needs substantial rewriting may not improve the work. Compare accepted drafts, significant corrections, unresolved exceptions and handling time against the baseline.
Avoid compressing every result into a single “accuracy” number. If routine drafts work well but requests involving expired quotations fail, the overall average can hide the exact boundary that needs attention. Examine results by scenario, and keep permission or duplicate-action failures visible separately.
Info
Human review is not a substitute for testing. Reviewers need a manageable queue and enough context to notice mistakes. Measure how the review actually works, rather than assuming a person will catch every problem.
Roll out with a way back
Test the workflow without sending messages or changing live customer records. Review incorrect outputs and check that failed cases reach the expected state.
Let the team compare drafts with what they would normally do. Keep responsibility for the real task clear so it is neither duplicated nor missed.
Start with the agreed kind of enquiry, sources and permissions. Keep a practical pause control and a manual path for the work if a dependency fails.
Repeat the relevant tests when prompts, models, source structures, permissions or connected tools change. Expand only after the owner understands the results and can support the additional scope.
Agree ownership for the day after launch
Record who maintains source documents, who investigates failed runs and who can change approval rules. Keep enough operational history to understand an action, while limiting access and avoiding unnecessary copies of sensitive content.
Agree which failures create an alert and where it goes. An error log that nobody reads does not give the team a recovery path. Also agree when a problem should pause the workflow and how unfinished tasks return to the team.
These decisions belong in the delivery scope alongside the software. Lindevo's approach to building and operating a system starts with the work, its boundaries and the people responsible for it. If you are still choosing the task, use our first-automation guide.
On this page