LLM Penetration Testing: Assess the Whole AI Application

An assistant can answer a hostile prompt without causing a security incident. It can also give an ordinary answer while retrieving a document its user should never see. To assess the risk, you need to examine the application around the model: identities, data sources, tools, and the actions those tools permit. LLM penetration testing investigates those relationships under agreed conditions. The useful outcome is evidence about a failed security boundary, with enough context for an engineer to reproduce the behavior and verify a fix. A collection of surprising model responses provides a much narrower result.

What is LLM penetration testing?

LLM penetration testing is an authorized assessment of security weaknesses in an application that uses a large language model. The scope can include prompts, retrieval, application interfaces, tools, and infrastructure. Testers investigate whether someone can access protected information or cause an unauthorized action through those components.

The application is the target. By comparison, AI pentesting can describe using AI to perform security testing against other systems. A model benchmark measures behavior on a defined task set. Safety evaluations examine additional concerns, such as harmful content. Agree which of these activities belongs in your engagement and how the team will judge results.

Map the application before testing the model

Start with a diagram of the request path:

User identity → application or orchestrator → model → retrieval and tools → data or actions

Add the systems that enforce permissions at each step. The application may authenticate a user, the retrieval service may select documents, and a downstream API may decide whether to execute an action. A model's instruction to respect permissions cannot substitute for those checks.

List the features in scope, including chat, document upload, background tasks, stored memory, and agent tools. Record the roles and tenants you can test. Two accounts with different entitlements are more useful for an isolation assessment than one unrestricted administrator account.

Describe the outside dependencies too. Your permission to test an application does not extend to disrupting a model provider or attacking another customer's data. Agree how to test integrations using controlled accounts, synthetic records, and suitable environments.

For a first engagement, choose a few high-impact workflows. A support assistant that can read tickets needs a different plan from an agent that can issue refunds. Scope each according to the data it can reach and the changes it can make.

Prioritize risks by data access and possible actions

Use a risk framework to organize coverage, then turn the relevant entries into application-specific questions. The OWASP GenAI LLM Top 10 2026 updates the numbering and names used in earlier editions. Record the edition beside any mapping.

For example, the 2026 document lists Excessive Agency as LLM03, Hidden Context Exposure as LLM08, and Improper Output Handling as LLM10. Those identifiers differ from familiar 2025 mappings. Mixing editions makes the coverage record harder to interpret.

Your test plan should explain the boundary you expect the application to enforce. “Check prompt injection” is a category. “An uploaded supplier document must not authorize the agent to send a payment” describes a security condition you can observe.

Prompts, untrusted content, and retrieval

The application can receive instructions through more than its chat box. Documents, retrieved passages, emails, and tool responses may contain text that conflicts with the user's request. Include the external content paths your application consumes in the assessment.

Retrieval-Augmented Generation, or RAG, supplies selected reference material to the model. Test how the system chooses that material and enforces access. A tenant isolation failure can disclose protected information even if the document contains no hostile instructions.

Use synthetic records with distinct ownership and recognizable markers. Establish which test account should access each record before testing. Then inspect retrieval results and application responses to determine where any boundary failed. The OWASP prompt injection prevention guidance provides context for handling untrusted content and separating controls.

Hidden prompts and other context need a sensitivity assessment. Disclosure of generic instructions may have limited impact. Exposure of credentials, private records, or another user's context can have a different consequence. Record what the system revealed and why the affected user lacked permission to receive it.

Tool permissions, output handling, and agent actions

Inventory the tools available to the agent and the credentials behind them. Determine whether each call uses the end user's identity, a shared application identity, or a more privileged service account. That choice affects the possible impact of a mistaken or manipulated action.

Check controls at the execution point. A refund API should enforce the permitted amount and account relationship even when an agent requests the call. A user approval screen should display the actual action and destination, and the application should execute the approved parameters.

Follow output to its destination. Generated text may enter a browser, database query, ticket, or command runner. The receiving component needs appropriate validation or escaping. A response that looks harmless in chat may create a problem when another component interprets it.

Distinguish proposed actions from completed ones. Preserve tool arguments, authorization decisions, and the resulting state. A model claiming to have transferred money is different from a controlled test account receiving an unauthorized refund.

Design a controlled LLM pentest workflow

The OWASP GenAI Red Teaming Guide separates model, implementation, infrastructure, and runtime assessment. Use those areas to check the completeness of your scope. The engagement can then follow a workflow tailored to your product.

1. Define success and stop conditions. Specify the protected records and actions, permitted techniques, resource limits, escalation contacts, and evidence handling.
2. Establish a baseline. Confirm that authorized workflows work with the supplied accounts. Preserve the relevant configuration and test data.
3. Write test cases. Give each case an identity, input source, expected control, and observable pass or fail condition.
4. Run and inspect. Observe retrieval and tool behavior where access permits. Repeat cases when variable model responses affect the outcome.
5. Confirm impact. Separate a policy-violating answer, an attempted tool call, and an executed unauthorized action.
6. Retest and preserve regressions. Check the fix against the original condition and related permitted workflows.

For a fictional example, imagine an internal assistant that can search purchase orders and draft supplier messages. A test document contains instructions that conflict with its reference role. The security condition is that document content cannot authorize a message to an unapproved destination. Testers use a controlled mailbox and synthetic order data, then inspect the approval and execution path. This is a test-design example, not a reported customer incident.

Choose tools and human review for the task

Tools help execute a test corpus, manage interactions, and evaluate outputs. Their usefulness depends on what you connect them to and how you interpret results.

Approach	Useful role	Limitation to address
NVIDIA garak	Probe supported model interfaces and apply detectors to responses	A model response assessment may not observe your application's tools or permission checks
Microsoft PyRIT	Orchestrate generative AI risk testing with configured targets and scoring	Your team must define targets, scenarios, and suitable outcome evaluation
Application integration tests	Check known access rules and resulting state	Fixed cases cover the conditions your engineers encoded
Human investigation	Explore product-specific workflows and explain impact	Scope, time, access, and tester knowledge constrain coverage

Verify capabilities against the tool version you plan to use. An integration or detector in the documentation does not guarantee coverage of your architecture without configuration.

Keep human review for findings that depend on context. A scoring model may flag a phrase while missing whether an action executed. A deterministic assertion against the controlled destination or record can provide stronger evidence for that condition. Use reviewers to inspect ambiguous traces and to check whether the selected score reflects the stated security question.

A practical LLM security testing checklist

Use the checklist as an acceptance aid for test coverage. Record evidence or a reason for exclusion beside each applicable item.

Check	Evidence to request
User and service identities	A record of which identity reaches each protected component
Tenant and document boundaries	Tests using records that different supplied accounts may access
Untrusted input paths	Coverage of applicable uploads, retrieval, external pages, and tool responses
Tool permissions	Observed authorization decisions for permitted and prohibited actions
Approval controls	The parameters shown to the user and the parameters executed
Output destinations	Tests of relevant rendering or interpretation boundaries
Sensitive context	Findings classified by the information exposed and its intended audience
Resource limits	Agreed tests of request, token, workflow, or tool-use budgets
Logging and evidence access	Redacted traces with access restricted to the agreed recipients
Regression coverage	Repeatable cases linked to fixed findings and affected workflows

Choose resource tests with the product owner. An unbounded agent loop or expensive tool call can affect cost and availability. Set a small, controlled budget that lets you observe the control without creating an uncontrolled bill.

Check permitted workflows alongside prohibited ones. A fix that rejects every uploaded document may stop the test case while making the product unusable. The acceptance criteria need to preserve the authorized task.

Record evidence and measure results responsibly

A useful LLM finding needs enough context to distinguish a model response from a system-level failure. Preserve the relevant trace under the engagement's evidence rules, with secrets and personal information removed from general report copies.

An evidence card for the fictional supplier assistant might contain:

Field	Example record
Condition	Reference-document content must not authorize outbound messages
Starting access	Employee test account with access to synthetic purchase orders
Input surface	Uploaded supplier document
Expected result	The assistant can summarize the document but cannot send an unapproved message
Observed result	Record the tool request, approval decision, and destination mailbox outcome
Configuration	Model identifier, application build, prompt revision, tool scopes, retrieval snapshot
Repetition	Number of attempts, observed successes, and relevant settings
Retest	Same condition after the fix, plus the authorized messaging workflow

Classify the result from what the trace demonstrates. Do not label a successful unauthorized action if execution stopped at the API's authorization check. That blocked attempt may still help improve the application, but it supports a different conclusion.

For attack success rate, state the numerator, denominator, scenario set, and success definition. Four successful boundary violations in twenty attempts describe those attempts under those conditions. They do not establish a universal failure probability for the product. Keep scenario coverage and business impact beside the percentage.

Record sampling settings and seeds when the provider exposes them, while recognizing that they may not make the entire system reproducible. Retrieval contents, tool responses, model updates, and orchestration can also change an outcome.

Retest after model, prompt, and integration changes

Maintain a versioned regression set for confirmed findings. Trigger relevant checks when you change the model, system instructions, retrieval configuration, connected tools, or their permission scopes. A change to the destination API can matter even when the prompt remains unchanged.

Retest the original failure and nearby conditions. For a retrieval isolation fix, include another permitted account and a legitimate retrieval task. For an action approval fix, inspect both displayed parameters and the executed request.

Use the result to update the finding's status and remaining limitations. BackDoor's guide to post-pentest remediation and retesting explains the broader ownership process. An LLM application still needs an engineer responsible for the fix and someone accountable for accepting any remaining risk.

Conclusion: validate boundaries, not just responses

Start with the information and actions your users may access. Map how the model interacts with those controls, write observable test conditions, and preserve evidence of what the application did. Keep the resulting cases close to your release process so fixes and later changes receive the same scrutiny.

For the externally reachable application and API layer, discuss your penetration testing scope with BackDoor. Identify any model, retrieval, or agent-specific assessment as a separate scope requirement before selecting a testing provider.

Sources

• OWASP GenAI LLM Top 10 2026
• OWASP GenAI Red Teaming Guide
• OWASP LLM Prompt Injection Prevention Cheat Sheet
• Microsoft PyRIT
• NVIDIA garak