Red-Teaming Enterprise AI: An Architectural Audit Checklist for the OWASP LLM Top 10

The case for AI red teaming an application, and not only its model, was made on 22 July 2025. Researchers at Cybernews found that a single 400-character prompt could turn Lena, Lenovo's GPT-4 customer-support chatbot, against the support site that hosted it. The prompt asked an ordinary product question, then told the chatbot to answer in HTML that loaded an image which did not exist. The site stored that answer and ran it. When the image failed, the browser sent its session cookies to the researchers' server, and the researchers reported the same result when a human support agent opened the conversation.^[1] Lenovo acknowledged the report on 6 August and said it had "implemented corrective actions".^[2] The researchers went no further, and neither report says an attacker ever used the flaw.

Nothing in that chain needed a jailbreak. The model did as it was asked, and the application around it trusted the result. Finding that kind of weakness before an outsider does takes more than a list of prompts.

This checklist is part of my enterprise guide to defending against AI-enabled attacks. It covers the evidence boundary of the Deterministic AI Boundary Model: can you reconstruct and prove what happened, and was it tested against a real attacker?

What should an AI red-team engagement cover?

Every entry in the OWASP Top 10 for LLM Applications, tested at the point where the model's output, retrieval or tool call meets a system that acts on it. For each entry the audit states the test, what the attacker knew, and the evidence that the control held or failed. A list of prompts the model refused does not do that.

The 2025 edition was still OWASP's current list in November 2026.^[3] Where the system has agents, add the OWASP Top 10 for Agentic Applications, released in December 2025, which maps its ten entries back to the LLM list.^[4] The table is my own working checklist. The tests and the evidence are derived from the entries and the research below.

Four of these rows have their own insights: indirect prompt injection in RAG pipelines, securing AI agent tool calling and tenant isolation in vector databases. The rest of this one covers the three that audits most often skip: output handling, the supply chain and poisoning.

How do you test for improper output handling?

Make the model produce every kind of active content the application could pass on: HTML, script, Markdown links and images, SQL, shell commands and URLs. Follow each one to where it is rendered, stored or executed, and check that the receiving system encodes it, parameterises it or refuses it. Test the screens staff use, not only the user's own.

OWASP's LLM05 is about what happens to model output downstream, and lists cross-site scripting and request forgery in browsers and remote code execution on back ends among the results. Its first mitigation is to "Treat the model as any other user, adopting a zero-trust approach".^[5] The Lenovo researchers named the same causes: output that was not sanitised, a web server that did not verify what the chatbot produced, and content loaded from arbitrary addresses.^[1] The second place the code ran, by their account, was the human agent's view of a stored conversation. A test of the customer's chat window alone would have missed it.

A content security policy is not the end of the test. In EchoLeak, the Microsoft 365 Copilot vulnerability that Aim Labs disclosed in June 2025, the final steps were an image the client fetched automatically and a request routed through a Microsoft Teams address that the policy allowed, according to Reddy and Gujral's case study. Microsoft fixed it and reported no evidence of exploitation.^[6] So the checks are:

• every destination of model output is listed: browser, email, ticket, log viewer, database, shell, HTTP client;
• output is encoded for each destination, and Markdown links and images are limited to listed hosts;
• the content security policy forbids inline script, and nothing on its allow-list will forward a request;
• SQL written by the model is parameterised or runs under a read-only role, and code it writes runs in a sandbox with no network;
• a canary address is embedded in the test output, and the test passes only if nothing ever calls it.

How do you assess AI supply chain risk?

List everything between the user and the model's output that someone else wrote or operates: model weights, adapters, API routers, MCP servers, packages and datasets. For each, record where it came from, how its integrity is checked when it is loaded or called, and what it could do if it were malicious. Then test the riskiest ones as though they were.

OWASP's LLM03 says plainly that "Currently there are no strong provenance assurances in published models", and recommends an inventory, signing and file hashes.^[7] The agentic list adds that agents assemble tools at runtime, so the inventory is not fixed at build time.^[4]

Model files. Lee and colleagues showed in 2025 that an unsigned malicious PyTorch model file ran a remote payload when loaded, because the file format can execute code, and that loading only signed, validated models blocked it.^[8] Scanning alone is weaker evidence. Researchers at Palo Alto Networks built TensorFlow models that read files and used the network through obscure features of legitimate functions; the model hub's scanner did not flag their example, because such scanners match known signatures.^[9] The test: load each third-party model in a sandbox with no network, and record what it tries to do.

Routers. An API router that sits between an agent and the model provider sees every request and can rewrite every response. Liu and colleagues bought 28 paid routers and collected 400 free ones in 2026; nine were injecting malicious code, and 17 touched the canary cloud credentials the researchers had planted.^[10] Ask who runs any intermediary in the path, and send a canary credential through it.

Tool descriptions. In MCPTox, instructions hidden in the descriptions of tools on 45 real MCP servers succeeded against one model in 72.8% of cases, and the most cautious model refused fewer than 3%.^[11] That is a benchmark, not a field measurement. The audit step: pin tool descriptions and compare them on every update.

How do you test for data and model poisoning?

Mostly by auditing, not by prompting. Behavioural testing cannot show that a model has no backdoor, because a backdoor is built to stay silent until its trigger appears. Establish who can write to training, fine-tuning and retrieval data, check the provenance records, and test whether a low-privileged contributor can plant content the model later relies on.

OWASP's LLM04 describes backdoors that "may leave the model's behavior untouched until a certain trigger causes it to change".^[12] The 2024 Sleeper Agents preprint shows why. Hubinger and colleagues deliberately trained models that wrote secure code when the prompt said the year was 2023 and exploitable code when it said 2024. The behaviour persisted through further safety training, and adversarial training, which is red teaming followed by retraining, could teach a model to hide the behaviour better.^[13] These were backdoors the authors built to study, not ones found in released models.

So a clean red-team run says little here. Lee and colleagues make the same limit explicit: their checks stop unsafe files, and do not prove that a model has no backdoor in its behaviour.^[8] So the evidence that passes is provenance: where each dataset and model came from, who could change it, and the version history. I found no public, sourced case of a backdoored model causing loss in a named deployment; the evidence so far comes from the laboratory.

Why are static prompt lists not enough?

Because a fixed list measures yesterday's attacks against today's defence. When researchers let the attacker see the defence and adapt to it, defences that had scored near zero on fixed benchmarks were bypassed much of the time. A red team should know the system's defences, iterate against them, and report the attempts as well as the successes.

Zhan and colleagues reported in 2025 that adaptive attacks bypassed all eight prompt-injection defences they evaluated, with attack success consistently above 50%.^[14] In the AutoDojo preprint (2026), a filter that allowed no attacks from a static set allowed 28% once a black-box attacker adapted.^[15] Narisetty and colleagues, in another 2026 preprint, argue that the newer deterministic defences have the same gap in their evidence: each was validated on a fixed set of attacks. Their protocol specifies an attacker who "knows the policy", and reports attack success alongside the cost to usefulness.^[16]

Their own small test went the other way. A deterministic privilege-control layer cut attack success from 25.8% to 4.2%, and a hand-built adaptive attack did not raise it. That was one weak open model and one attack family; the authors say it does not establish that deterministic controls are harder to beat.^[16]

Real attackers do reconnaissance first. Xie and colleagues red-teamed six coding agents by extracting the system prompt and tool descriptions, which worked on 19 of 25 agent and model pairs, then using what leaked to build the attack. They reported remote code execution on every pair they tested, as at 2025.^[17] OWASP's position is that a system prompt "should not be considered a secret, nor should it be used as a security control".^[18] Give the red team the prompt on day one.

My view: a refusal rate is a fact about a model on a day. An audit should produce facts about the system: which controls sit outside the model, and what happened when someone who understood them tried to get past.

What evidence should every finding include?

Enough for someone who was not there to reproduce it and to check it: the exact build and model tested, what the attacker knew at the start, the steps, the number of attempts, and the raw requests and responses with times and hashes. It also draws a clear line between what was demonstrated and what is inferred.

Level 3 of the boundary model, Verified, requires testing against adaptive, defence-aware attackers and logs that reconstruct every decision. A finding record is the test's half of that. This is a template; the values are placeholders, not results from a real engagement:

{
  "id": "F-014",
  "title": "Model output rendered as HTML in the staff console",
  "owasp": ["LLM05:2025"],
  "boundary": "action",
  "level_observed": 1,
  "system_under_test": {
    "build": "support-chat 4.12.3",
    "model": "provider/model-id@2026-10-01",
    "system_prompt_sha256": "<hash>",
    "tools": ["lookup_order", "handoff_to_agent"]
  },
  "attacker": {
    "role": "anonymous customer",
    "knowledge": "system prompt and tool list disclosed to the tester"
  },
  "attempts": 12,
  "successes": 9,
  "steps": ["<numbered steps, payload stored by hash>"],
  "evidence": [
    { "type": "http_exchange", "file": "f014-03.har",
      "sha256": "<hash>", "captured_utc": "2026-11-03T02:14:55Z" }
  ],
  "demonstrated": "Tester's own session cookie sent to the canary host",
  "not_attempted": "Use of a staff session; access to customer records",
  "control_expected": "Output encoding at the console; CSP without inline script",
  "retest": { "date": null, "result": null }
}

The fields that matter most are demonstrated and not_attempted. The Lenovo researchers kept them apart: they captured session cookies, and of anything further they wrote, "We didn't attempt any of this".^[1] Some headlines still called it a breach; neither report I read describes one.^[2] A record that separates the two protects an organisation from understating a finding and from overstating it.

When a finding is later disputed, by a vendor, an insurer, a regulator or in court, it is tested the way any evidence is. Can it be reproduced? Was the record made at the time? Does the conclusion go further than the observation?

My view: write every finding for the reader who wants it to be wrong. If the build, the attempts and the raw exchange are on the page, that reader has nothing to argue with except the fix.