Is It Safe to Let an AI Agent Send Email or Pay Invoices?

The technical detail: Securing Autonomous AI Agents: Deterministic Gateways for Tool Use and Function Calling

In April 2025, customers of Cursor, a software tool for programmers, found themselves logged out when they switched between devices. Those who wrote to support received an email from "Sam" saying this was expected under a new login policy. Fortune reported that there was no new policy and no person behind the email: an AI support bot had written it. Customers cancelled subscriptions, and a co-founder publicly acknowledged an incorrect response from the bot.^[1]

Most AI agent risks look like this one. Nobody attacked the business. It had let software send email to customers in its name, and the software was wrong. An agent that can also change records or pay invoices can be wrong in a way that has already happened by the time you find out.

This article is the plain-English part of my guide to defending an organisation against AI-enabled attacks.

Is it safe to let an AI agent send email or pay invoices?

Only within limits that you set and that the system enforces. An agent acts on the text it reads, and text can be planted or misread, so the useful question is what the most damaging thing it is permitted to do is.

An AI agent is an AI assistant that has been given tools: the ability to send, book, update, delete or pay.

What can go wrong?

Two things. The agent can be talked into an action by something it reads, or it can make an ordinary mistake while holding more power than the job needed. Either way the result is a real action: an email sent, a record changed, a payment made.

The Cursor email was the second kind: an ordinary mistake, sent to customers as fact. A hypothetical example of the first kind: an agent that looks after the accounts inbox receives an email that appears to come from a regular supplier. In it is a line addressed to the assistant: our bank details have changed, pay the outstanding invoice today. Whether the money moves depends on what the agent may do without asking.

What has the research measured?

In tests, agents with real tools and no added protection followed planted instructions most of the time, and agents reached for more power than a task needed with nobody attacking them. These results show what is possible, not how often it will happen in your business.

In a 2026 peer-reviewed study, researchers connected agents to real tools with real permissions and planted instructions in what the agents read. With no defences in place, the planted instructions succeeded between 79% and 91% of the time on average, depending on how the agent was built, across four current AI models. The authors describe these results as preliminary.^[2]

In 544 simulated tasks where a less powerful tool would have done the job, eleven AI models chose the more powerful tool between 2.6% and 64.9% of the time.^[3]

These numbers were measured on particular AI models in test settings, and the second study is not yet peer reviewed.

Does asking a person to approve each action solve it?

Not on its own. Approval works when it is rare and the person can see exactly what is about to happen. An agent that asks about everything trains people to click yes.

Researchers at Microsoft tested an agent that asked for approval before every action with consequences. It needed between one and nearly four approvals per task, and planted instructions still diverted it in 15% to 61% of cases, depending on the model and website. The same researchers note that frequent approval requests lead people to stop reading them, a point they take from wider research on repeated prompts and did not measure in agents.^[4]

What held up better in tests was a fixed rule enforced outside the AI. Researchers reported that when rules about which tools could run, and with what values, were enforced by ordinary software, successful attacks fell from about 40% to 1% on one standard test and from about 70% to 4% on another. The limit, they say: rules cannot stop misuse that stays inside what the task genuinely needs.^[5] This is the idea behind my Deterministic AI Boundary Model: decide what the AI may do, and enforce it outside the AI.

My view: an approval button is worth having only if the system decides when to ask, and shows the person the actual action, not the agent's description of it.

Which actions should an agent never take alone?

Anything that moves money, changes who gets paid, leaves the organisation, or cannot be undone. Let it read, draft and do routine internal work alone, so the approvals you ask for are few enough to be read.

The table is my judgement, built on advice from OWASP, an industry body for application security, to require a person's approval for high-impact actions.^[6] The support email that opened this article belongs in the third row: it left the organisation with nobody checking it.

What should you set up before you switch it on?

Cut the agent's powers to what the job needs and set limits the system enforces. Choose the few actions that need approval, and keep a record the agent cannot change.

1. List what it can do, not what you plan to use it for. Switch off every action the job does not need.
2. Give it its own account. Not yours, and not one shared with other agents, with the least access that works.
3. Set limits in the system. A maximum amount, a list of approved recipients, a daily count. A limit written in the agent's instructions is only a request.
4. Choose the actions that need a person. Use the table above, and keep the list short.
5. Check what the approver sees. It should be the real action: the recipient, the amount, the account number.
6. Leave "run without asking" off. If the product has a setting that skips approvals, find out who can turn it on.
7. Keep a record outside the agent. What it was asked, what it did, who approved it, and when.
8. Have it tested, and tell your staff. Ask someone to try to make it do something it should not. Tell staff a change of bank details is always confirmed by phone.

What would you need to show afterwards?

If a payment or an email is disputed, you will be asked who authorised it, on what information, and whether the record can be trusted. "The AI did it" answers none of these. A record kept outside the agent can.

Think of the support email again. The first job afterwards is to establish what was sent, to whom, and on whose authority. Where the only record of what an agent did is the one the agent wrote, a faulty or manipulated agent can leave things out or get them wrong.^[7] And where several agents share one login, the record cannot say which of them acted.^[8] Both points come from design analyses, not from observed incidents.

My view: as an expert witness on fraud and data manipulation, I look at what an organisation can show happened, not at what its policy said should happen. Pick one action your agent took last week and try to answer three questions from the records alone: who or what approved it, what the agent had read when it acted, and whether the record could have been altered since. If you cannot, the agent has more authority than you can account for.