Start with a release decision
A useful demonstration shows that an agent can complete a task. A deployment review needs to establish what else it can do, what could mislead it and who can stop it. Those questions become more important when the same system can read confidential records and change operational systems.
Use this checklist for one named workflow and one identifiable version of the application. Give every item an owner and supporting evidence. A statement such as ‘the agent asks for approval’ should lead to a test showing that an unapproved action is rejected by the application. Record any remaining gap and the operating restriction that keeps it contained.
1. Draw the agent’s authority boundary
Write a short description of the permitted job. For an alert-enrichment agent, that might mean reading an assigned alert, consulting approved intelligence sources and preparing a draft for an analyst. Closing the alert, disabling an account or contacting a customer are separate decisions.
Then draw the path through the model, tools, credentials, data stores and external services. Include scheduled jobs and any secondary agents. Mark where information crosses between customers, departments or environments. This makes the review specific enough to challenge: a reviewer should be able to point to each connection and explain why the workflow needs it.
- List permitted inputs, outputs and tool operations.
- Identify the person accountable for the workflow.
- Record the actions excluded from this release.
2. Prove permissions in the connected systems
Inspect the credentials behind every tool. A tool described as a reader may still authenticate with an account that can edit or delete records. Remove unnecessary functions and enforce access in the downstream service. Where work is performed for a user, preserve that user’s authorization scope. OWASP identifies excessive functionality, permissions and autonomy as distinct causes of excessive agency. [1]
Ask for a negative test: can the integration retrieve another customer’s record or perform a prohibited write? The test should demonstrate a denial by the authorization layer, even when the model requests the action.
3. Test instructions hidden in ordinary content
Documents, web pages and tool responses can contain instructions designed to redirect an agent. OWASP treats this as indirect prompt injection and warns that retrieval-augmented generation does not eliminate the problem. Separate external content from trusted instructions, and keep authorization independent of the text the model reads. [2]
Prepare harmless adversarial samples in the formats the workflow actually handles. Test an instruction embedded in a retrieved document, a misleading tool response and a request to send information to an unapproved destination. Record both the answer and attempted tool calls. A reassuring final answer is insufficient if an unauthorized request was attempted or completed earlier.
4. Make approval a real control
Require explicit approval for consequential actions, enforced outside the model’s discretion. OWASP recommends human approval and downstream authorization checks for this purpose. [1] The reviewer should see the exact destination, affected records and proposed change. If those details change, require a fresh decision.
Test rejection, expiry and cancellation. Confirm that a later retry cannot reuse approval for a different action. An approval screen is only useful when the execution path respects its result.
5. Validate what leaves the model
Treat generated output as untrusted input to the next component. Validate tool arguments against a narrow schema and permitted values; use parameterized database operations and context-appropriate output encoding. OWASP’s improper-output-handling guidance explains why apparently useful model output can become an application vulnerability when passed into browsers, commands or databases without checks. [3]
Test malformed identifiers, unexpected fields and destinations outside the permitted set. For a proposed ticket update, validate the ticket and allowed fields before applying anything. Keep the original record available so a reviewer can compare the proposed change with its source.
6. Put limits around every run
Set enforceable limits on execution time, tool calls, retries, queued work and resource consumption. Apply user or workload quotas where appropriate. OWASP’s unbounded-consumption guidance recommends quotas, timeouts, monitoring and limits on queued actions to reduce resource-exhaustion risk. [4]
Exercise the limits deliberately. A slow dependency should produce a controlled stop and a clear status. A repeated failure should not create an endless retry cycle. For actions that modify another system, check how the application determines whether a timed-out operation already succeeded before attempting it again. Assign someone to receive and investigate limit alerts.
7. Evaluate the complete workflow
Test realistic tasks, missing evidence, conflicting records and the security cases above. Separate answer quality from unauthorized-action attempts and data-boundary failures. NIST’s Generative AI Profile emphasizes contextual pre-deployment evaluation and sharing results with those responsible for release approval. [5]
Define acceptance criteria before reviewing results. Repeat important cases: one successful run is weak evidence for variable behaviour. Keep the model, prompts, tool definitions and application version with each result. A change to any of them should trigger the relevant checks again, including cases that previously failed.
8. Rehearse observation and shutdown
Give operators a record of tool requests, authorization decisions, approvals and outcomes, with sensitive fields minimized and access controlled. They need to reconstruct what happened and identify the affected records. NIST includes monitoring, override, incident response and recovery in post-deployment planning. [5]
Run a shutdown exercise. Disable the workflow, stop queued activity and verify whether credentials need revocation. Check the resulting state in connected systems. Name the person who decides when service can resume and the evidence they will require.
Bring a small release pack to the decision
The final review should contain the workflow boundary, permission inventory, test results, unresolved findings and operating instructions. For each exception, record an owner, a restriction and a review date. Avoid combining these into a single reassuring score that hides a failed access-control test.
Choose the next deployment step from that evidence: a limited pilot, a narrower scope or more remediation. Expansion should follow demonstrated control over the current workflow. That gives the business a concrete basis for deciding which additional data and actions the agent can be trusted to handle.
