How an AI and LLM pentest is run: the phases, test cases and what the report has to show

What separates LLM testing from a web application pentest

The input surface of an LLM is not a form field. A web application pentest works from a defined set of request parameters, roles and state transitions. An LLM system accepts natural language, and that language is both the communication channel and the attack channel. The same string that asks a question can reframe the model's instructions, override its system prompt, or coerce it into calling a tool it was told to leave alone.

The boundary that a web application pentest tests is between the caller and the application. The boundary an LLM pentest tests is between the caller and the model's context window, and between the context window and everything the model can reach: tools, APIs, file systems, downstream services. That second boundary is wider. Most test programmes do not cover it. The OWASP Top 10 for Large Language Model Applications maps the vulnerability classes; the methodology below maps how to test them.

Mapping the attack surface before testing begins

Before any payloads are sent, the tester needs a complete picture of the system. A scope document for an LLM pentest names the model, the system prompt and any instructions that sit outside it, the tools or plugins the model can call, the data sources it reads at inference time, and the output channels: chat UI, API, email, downstream service call. The article on how to buy an AI and LLM pentest covers what to prepare for that conversation. Naming only the API endpoint leaves the most significant surfaces uncovered.

From that inventory, the tester builds a threat model. Each tool the model can call is a potential pivot point. Each data source is a potential indirect injection vector. Each output channel is a potential path for extracted data to leave. The threat model records each component and the trust the model places in it: does the model treat the content of a retrieved document as instructions? If so, that document is an injection surface.

The tester also records what the model is authorised to do. That baseline matters. A finding only stands if the model acted outside its authorised scope, and a clear scope agreement fixes that boundary before the test starts, not after.

The four test phases and what each one covers

Reconnaissance is the first phase. The tester maps what the model knows about its own configuration. System prompt extraction does not require a vulnerability: it requires persistence. A sequence of probing questions, boundary tests and continuation prompts can recover large parts of a system prompt the operator intended to keep private. The tester records any configuration detail the model reveals, because those details inform later test cases.

Direct prompt injection follows. Every input the attacker controls is a direct injection surface, not only the primary chat field. A model that accepts a user-supplied document name, a support ticket subject line or a code snippet opens up surfaces outside the standard chat flow. The technical testing guidance in NIST SP 800-115 sets the principle: test every input, not only the obvious one.

Indirect prompt injection is the third phase. The tester inserts a payload into a data source the model is expected to retrieve: a web page, a document in the retrieval store, a database field, a calendar entry. When the model fetches and processes that content, the payload attempts to change what the model does next. The fourth phase covers agent abuse: whether a successful injection can make the model call a tool it was not directed to call. The article on UAE rules for AI and LLM security testing covers how regulators in the region are beginning to frame those risks.

The core test cases for prompt injection

Direct injection test cases fall into three families. Instruction override tests whether injected text can replace or extend the system prompt's instructions, using phrases such as "Ignore previous instructions" or continuation prompts that append new directives. Privilege escalation tests whether an injected role claim causes the model to reveal information or take actions it would otherwise refuse. Output shaping tests whether the attacker can alter the format or content of the response in a way that bypasses downstream validation. MITRE ATT&CK catalogues technique families across adversary behaviour; the relevant classes map to all three.

Indirect injection test cases depend on the retrieval architecture. For a model that reads from a web browser tool, the tester places a payload in a page the model is likely to retrieve. For a model that reads from a document store, the tester inserts a payload into a document the model is asked to summarise. For a model that reads from a database, the tester inserts a payload into a field the model queries. In each case, the payload attempts to change what the model does next: exfiltrate data, call a tool, modify output or relay information to an external endpoint.

The table below organises the five primary test case categories across both injection types, mapping each to its injection vector and what each test measures.

Agent chains and the blast radius of a successful injection

An LLM with access to tools is not only a chatbot. It is an automated agent that can read files, write files, call API endpoints, send email, execute code or query databases, depending on what tools it has been granted. The blast radius of a successful prompt injection in an agentic system covers everything the model is authorised to do, because a successful injection can redirect that authorisation toward the attacker's objective.

The test cases for this phase work from the tool list. For each tool, the tester constructs an injection payload that directs the model to call that tool in a way it should not: to exfiltrate a file, to call an API with attacker-controlled parameters, to relay data to an external endpoint, to modify a record. The test is not whether the model can call the tool; it can. The test is whether the model's safety checks can be bypassed via injection to make the call on behalf of the attacker.

Not every LLM system has agents. The AI and LLM pentest covers both stateless and agentic deployments, and the scope agreement distinguishes between them before the test starts. An agentic system requires at least one test case per tool in the model's tool list, because every tool is a potential pivot.

How findings are structured in an LLM pentest report

An LLM pentest finding shares its core anatomy with a standard penetration testing report: title, severity, evidence and reproduction steps. Two additional fields carry most of the remediation value. The first is the injection vector: the exact input the tester controlled and how it reached the model's context window. The second is the trust boundary crossed: which guardrail, instruction or authorisation check the injection bypassed. Without both fields, the developer reading the finding cannot tell whether the fix is to harden the system prompt, restrict the input surface, add output filtering or change the retrieval architecture.

Severity in an LLM finding scales with the authorised blast radius. An injection that overrides the system prompt in a stateless chatbot with no tools is lower severity than the same injection in an agent with write access to a production database. The article on what a penetration test report has to contain covers the baseline structure; an LLM report adds the two fields above. Retesting a finding means running the exact reproduction steps against the patched system and recording whether the injection still executes or the trust boundary now holds.

AI and LLM security testing in the UAE

The UAE has moved quickly to adopt generative AI across financial services, government and healthcare. In Dubai, the Dubai Digital Authority has published an AI governance policy that frames responsible deployment of AI systems. In Abu Dhabi, the Technology Innovation Institute conducts foundational model research. Neither document is a mandatory pentest requirement, but both show that regulators in both emirates are watching how those systems handle data and what access they hold.

For regulated entities, existing frameworks apply. CBUAE's guidance covers any system a bank or payment service provider deploys, including AI components that touch customer data or automated credit decisions. DESC binds Dubai government entities and their suppliers: a contractor integrating an LLM into a government-facing service sits within DESC's scope. ADHICS applies to Abu Dhabi healthcare providers who use AI for clinical documentation or patient-facing tools. The NESA Information Assurance Standard applies federally to designated critical information infrastructure operators. Any organisation in those categories that deploys an LLM without a security assessment is running a system whose risk profile has not been reviewed before it handles real data.

Frequently asked questions