How to assess whether an application is ready for AI agents

Expected outcome

At the end of this assessment, the team has a one-page decision record for a single application workflow: approve a bounded pilot, approve only an assistive mode, or defer the work. The record names the initiating user, the data that can enter the workflow, the permitted tools and actions, the acceptance checks, the operator who handles failure, and the condition that stops the pilot.

This is not a vendor scorecard. It is an application control exercise. A product can document graph state, tool calling, or authorization while the application still lacks a safe way to carry a user identity, prevent duplicate updates, delete retained data, or explain why a task was completed. The agent runtime platform selection guide helps compare frameworks after those operating needs are visible; it cannot replace the evidence record itself.

Use the result to reduce ambiguity before a model can touch production data or an external action. A good first candidate is a read-only classification, extraction, drafting, or recommendation workflow with a human reviewer. A poor first candidate changes entitlements, sends irreversible communications, or commits a business record without a deterministic authorization and recovery path.

Prerequisites

Bring one representative business task, not a broad aspiration such as “add an agent to support.” Name its primary user, the decision it helps make, the systems it reads, the systems it could change, and the outcome that a reviewer can verify. Obtain a non-production environment or a reversible test surface. Do not begin with a production service credential shared by every user.

You also need four people or roles, which may be held by a small team: a business owner who accepts the outcome, an application owner who can change the system, a security or privacy reviewer for the data boundary, and an operator who receives alerts. One person may initially fill more than one role, but the decision record should make that visible rather than implying that “the agent” owns a failure.

Prepare a small versioned evaluation set. Include ordinary requests, incomplete data, stale or contradictory information, a user who lacks permission, an untrusted document with malicious instructions, a dependency timeout, and a request that must be refused. The agent workflow guide is useful for turning those cases into observable inputs, stages, reviews, and outputs. The set does not need to be large; it must expose the conditions under which the application should not proceed.

Step 1: Write the task and boundary card

Describe the task in a short card with six fields: initiating identity, allowed input sources, expected output, permitted tools, prohibited actions, and safe failure behavior. Write the expected output as an acceptance contract, not a vague request. For example, an intake assistant may produce a typed draft and evidence links, while only a human can submit the final case to a downstream system.

List every boundary crossing. Mark where information enters the model context, where a service credential is used, where an API write may occur, and where a result is stored. Treat retrieved pages, issue text, attachment content, tool responses, and previous conversation state as untrusted until deterministic controls validate their use. This makes prompt injection a data-flow question instead of a generic warning.

For MCP-connected workflows, distinguish the end user, the client, the server, and the authorization server. The official protocol describes authorization for restricted HTTP MCP servers on behalf of resource owners, including resource binding and scope behavior. That supports a readiness question—“which resource and minimum scope are actually needed?”—not an assumption that a protocol feature automatically secures every application. Use the MCP security checklist to turn that question into concrete tool allowlists and approval rules.

Step 2: Prove identity and least privilege

For each tool, record the identity used at runtime, the role or scope it receives, the resource it can reach, its expiry, and its revocation path. A service token may be necessary for a background worker, but it should not silently grant every requester the same access. Test two users or tenants with different permissions against the same request. Verify that denied data is absent from the model context, tool response, trace, cache, and final output.

Ask whether the tool needs read, draft, write, delete, or administrative capability. Start with the narrowest option. Require a deterministic server-side authorization check immediately before a consequential action; a model’s decision or an approval-looking message is not an authorization system. If an action can be repeated after a retry, attach a stable operation identifier and record completion in the business system of record.

Secrets deserve their own inventory. GitHub’s official documentation distinguishes repository, environment, and organization secret handling and warns that redaction is not a reason to pass sensitive data broadly through logs. Keep deployment secrets out of prompts, test fixtures, error messages, and generated reports. Record which secret manager, workflow environment, and human role may change each credential.

Step 3: Map data, retention, and evidence

Make a field-level inventory: prompts, uploaded files, retrieved passages, embeddings, tool arguments, tool results, traces, user feedback, logs, backups, and support exports. For each field, record purpose, classification, reader role, storage location, retention period, deletion method, and whether it may leave the intended region or tenant. “We use a private repository” is not a data-flow answer.

Then decide what evidence an operator must see. A useful audit event contains a request ID, application and policy version, user or service identity, allowed tool name, normalized arguments, approval decision where required, result reference, error code, and timestamp. Avoid logging raw secrets or unrestricted user content simply to make debugging easier. A trace should support investigation without becoming an uncontrolled second data store.

NIST’s AI RMF provides a risk-management frame for governing and measuring AI systems; apply it here by connecting each identified harm to an owner, a measurement, and a response. A concern without a signal or response belongs in the deferred-risk list, not in an approval statement.

Step 4: Run the controlled evaluation

Freeze the model, prompt, tool definitions, policy configuration, and evaluation cases for the first run. Exercise the expected path, but give more weight to denial and recovery cases. Remove an upstream dependency, provide stale data, send a malformed tool result, try an out-of-scope argument, interrupt after a side effect succeeds, and wait past an approval deadline. Each scenario should produce a visible, safe state rather than a silent retry loop.

Record output quality separately from action safety. A typed object may validate while a factual assertion remains unsupported; a helpful draft may still be unacceptable for a regulated workflow; a successful API response may hide a duplicate change. Define task-specific checks such as evidence requirements, schema validation, authorization tests, policy denials, idempotency checks, latency budgets, and reviewer disposition.

If a reviewer changes an output, classify the reason: missing evidence, wrong data, wrong policy, unclear presentation, unauthorized action, or operational failure. That classification is more useful than a generic satisfaction score because it identifies whether to improve data, prompts, rules, training, interfaces, or the workflow boundary.

Step 5: Decide the operating mode and owner

Use three outcomes. Approved assistive pilot means the application produces drafts or recommendations and a human retains the final action. Bounded automated pilot means a narrow reversible action has deterministic authorization, explicit idempotency, monitoring, and an on-call owner. Deferred means a mandatory control is missing, so the team documents the gap and selects a smaller task instead.

Set a pilot expiry date, user population, allowed data classes, allowed actions, budget, and review cadence. Do not convert a successful internal demo into open-ended production authority. The agent approval workflow guide can help specify who may approve a high-impact result and what evidence the approver needs; it is not a substitute for the downstream service enforcing the decision.

Copyable application-readiness record

Copy this template into the team’s change record and fill it with evidence links rather than screenshots alone:

Workflow and owner:
Initiating identity and tenant boundary:
Allowed inputs and prohibited inputs:
Expected output and measurable acceptance checks:
Permitted tools, scopes, and downstream resources:
Prohibited actions and safe refusal behavior:
Data inventory, retention, deletion, and trace readers:
Secrets, rotation owner, and revocation test:
Evaluation cases and results, including denials and outages:
Idempotency key, retry limit, timeout, and recovery state:
Approval requirement and server-side enforcement point:
Monitoring signal, alert receiver, and incident owner:
Pilot users, expiry date, budget, and rollback trigger:
Decision: assistive pilot / bounded automation / defer
Accepted limitations and next review date:

The useful standard is completeness, not polished prose. If a field cannot be filled with an owner and evidence, it identifies work that must happen before the application receives broader authority.

Failure modes

• A prototype succeeds with a privileged developer account, then fails to preserve ordinary-user permissions in the deployed service.
• A background worker retries after a timeout and repeats an external update because there is no operation record.
• A trace makes debugging easy but retains sensitive prompts, source files, or secrets longer than the application policy permits.
• A review button exists in the interface, while a direct API or queue path can bypass the same approval requirement.
• A team measures fluent outputs but never tests stale data, prompt injection, denied access, dependency outages, or an abandoned approval.
• An owner changes models, tools, or retention settings without rerunning the acceptance set or recording the change.

When one of these appears, do not compensate with a longer prompt or a more persuasive demo. Reduce the task, close the missing boundary, add a test that proves the control, and rerun the decision record.

Acceptance criteria

Approve a pilot only when all of the following are true:

1. One named workflow, accountable business owner, application owner, and incident owner are recorded.
2. The evaluation set includes an expected case, a denial, untrusted content, a dependency failure, and a recovery scenario.
3. Every tool has a documented runtime identity, minimum scope, allowed resource, and revocation path.
4. Consequential actions have deterministic authorization, an operation identifier, a retry boundary, and a visible terminal state.
5. Data, traces, and backups have named readers, retention, and deletion procedures appropriate to the workflow.
6. The pilot is limited to a defined user group, allowed data, reversible or reviewable actions, budget, duration, and rollback trigger.

Next step

Run the assessment for one read-only or human-reviewed workflow, then hold a short review with the named owners. If the record is complete, schedule a limited pilot and keep the evaluation corpus fixed long enough to compare changes. If it is incomplete, pick the highest-risk missing field—often identity propagation, recovery, or trace retention—and solve that before adding another tool or model.

Sources were publicly checked on August 2, 2026. They establish protocol, secret-handling, and risk-management references, not a guarantee that any particular application is ready. This tutorial does not report hands-on testing or a security certification. Recheck current documentation, terms, privacy requirements, and target-environment controls before a production decision.