At What Point Should Humans Still Check AI and Automation Results

Once AI or automation is actually running in a business, one question always comes up: can this run on its own, or should someone check it first? If the answer is "everything must be checked", the automation is pointless. If the answer is "no need to check", you're just waiting for an incident.

The answer sits in the middle, and what decides it isn't gut feeling but risk. Two stories show why.

Two stories about what happens when nobody checks

A chatbot gave wrong information, and the company was held responsible

In February 2024, a tribunal in British Columbia ruled in Moffatt v. Air Canada (2024 BCCRT 149). Air Canada's chatbot told a passenger he could apply for a bereavement fare retroactively after flying, even though another part of the airline's site said the opposite. Air Canada argued the chatbot was effectively a separate entity. The tribunal rejected that argument. According to McCarthy Tétrault's summary, the ruling says: "It makes no difference whether the information comes from a static page or a chatbot." A company is responsible for what its chatbot says, just as it is for its web pages.

An AI agent that deleted a production database

In July 2025, Jason Lemkin, founder of SaaStr, reported that Replit's AI agent deleted his production database even though he had repeatedly instructed it to follow a code freeze. According to The Register, the service admitted a "catastrophic error of judgement" and said a rollback was impossible, which later turned out to be wrong because the rollback worked. The agent also created fake data at one point. In Lemkin's words, "There is no way to enforce a code freeze in vibe coding apps like Replit."

The lesson: an instruction inside a prompt is not a fence. The fence has to live in the system, through permissions that simply don't allow the action.

The level of oversight follows risk, not feeling

OpenAI's A practical guide to building agents names two triggers that usually justify bringing in a human: the agent exceeds a failure threshold (for example, it fails to understand a customer's intent after several attempts), and high-risk actions, meaning ones that are sensitive, irreversible, or high-stakes, such as canceling orders, approving large refunds, or making payments. The guide also suggests rating each tool an agent can use as low, medium, or high risk, based on whether it's read-only or can write, whether it's reversible, the access permissions it needs, and its financial impact.

Anthropic, in Building effective agents, writes that an agent can pause to ask for human input at certain checkpoints or when it hits a blocker, and warns that the autonomous nature of agents means higher costs and the potential for compounding errors.

From there, I use a simple split that's easy to apply in a business:

An approval that's only a formality is the same as none

This is the part that often gets overlooked: putting a human in the middle of a process doesn't automatically make it safe. A review by Parasuraman and Manzey in Human Factors (2010) on complacency and bias in automation concluded that automation bias can make people err both by missing things and by following wrong advice, that it affects novices and experts alike, and that it cannot be prevented by training or instructions.

So saying "please check this" isn't enough. What needs designing is the process. A few things I think help (these are my suggestions, not findings from the review above):

• Show what changed or looks unusual, not just "is everything correct?".
• Limit the volume. If there are 100 approvals a day, people will click through without reading. Use thresholds (amount, type, confidence score) so humans only see what needs it.
• Build an exception queue. What matches the usual pattern runs on its own, and what's odd goes into the queue.
• Randomly sample what passes, so errors that aren't visible still get caught.
• Log who approved what and when.
• Limit access from the start. An agent without delete rights on production can't delete production, however smart it is.

Examples from systems I've built

• The finance system for a kos, the Indonesian take on a boarding house where tenants rent furnished rooms by the month. Invoices are created by an admin and then approved (all of one tenant's invoices can be approved in a single click), incoming proof of payment goes to the management team to verify, and the receipt is only sent after the payment is verified. Money moves at every step, so humans sit at the approval and verification points. Due-date reminders, which are reversible and low risk, run on their own. The details are in the boarding house finance system case study.
• WhatsApp marketing recap. AI never steps into customer conversations. The two marketing team members still do the replying, and the AI only summarizes for the owner. That was a deliberate design decision, and I cover it in the WhatsApp Marketing Recap case study.
• Document verification. AI reads and validates documents, then one admin runs the system and checks the results. The cost dropped a lot compared with the four admin roles before, but the human is still there, as in How I Cut Operating Costs From Tens of Millions of Rupiah to Hundreds of Thousands.

Deciding the level: five questions

1. Can it be undone? If not, raise the level.
2. Who sees the result? Internal only, or outsiders such as customers, vendors, the public.
3. How bad is the worst case? In money, reputation, and legal exposure.
4. How often does the AI get this task wrong? Don't guess. Try it on real examples first and look at the results.
5. How fast would a mistake be noticed? A mistake found a week later costs far more than one caught immediately.

Start at level 2 for anything new, then move down to level 1 once your spot-check log has been clean for a few weeks. OpenAI's guide above also says human oversight for high-risk actions is worth keeping until confidence in the agent's reliability grows.

Wrapping up

So where should a human still check? At actions that can't be undone, that outsiders see, and that involve money or people's data. Let everything else run on its own, with samples checked regularly. For more on when a system really needs an agent that acts on its own, see Agentic AI: When You Actually Need It, and Platforms You Can Run Directly on a VPS. And on the risk side of tools that build apps from prompts, there's You Can 'Code' With AI Now, But Do You Still Need a Real Developer?.

If you're designing an automation and aren't sure which parts can run on their own, tell me about the process first, and I'll help map out the levels.

The cover photo for this article was made with AI, not a photo of a real person or place.