An AI agent can decide that an action appears necessary, but that does not mean the action should be allowed.
In a production agent harness, permissions, action limits, human approval, and cancellation form the control layer between a model's proposed action and a real-world side effect.
That distinction becomes important the moment an agent can do more than produce text. Reading an order, changing an order, issuing a refund, deleting a resource, sending a message, or modifying a production system all have different consequences. A good harness does not give the model one giant "permission" switch. It turns authority into narrow, inspectable decisions.
This article develops that idea through one fictional support-operations agent. The scenario is invented for teaching. The architecture patterns are general, while vendor-specific capabilities are identified where they are used as examples.
🍊 Child-friendly analogy
Think about a child using a family toolbox. Being able to see the screwdriver does not mean the child is allowed to use every tool, open every box, or repair the electrical panel. The harness is the set of rules that decides which tool can be used, on what, for how much, and whether an adult must approve it first.
Why this matters
OWASP's guidance on excessive agency describes the problem in terms of excessive functionality, excessive permissions, or excessive autonomy. It also recommends enforcing authorization in the downstream system instead of relying on an AI system to decide for itself whether an action is allowed. OWASP GenAI Security Project
🟢 Worked example: the motivating scenario
Imagine a support agent handling an order problem. It can look up an order and calculate a refund recommendation automatically. Cancelling the order changes customer-visible state. A large refund has financial consequences. The agent may therefore be allowed to propose those actions, while the harness decides whether they are permitted and whether a person must approve them.
📑 In This Post
- What permissions mean inside an agent harness
- Model choice is not authorization
- Designing action limits
- Human approval without approval fatigue
- Cancellation: stop the run, stop the action, or undo the effect?
- Build a durable control loop
- The complete running example
- Failure modes and Safety Checks
- Enterprise Rollout
- Common Mistakes
- ❓ FAQ
- 🔗 References & Further Reading
- 📝 Summary
🔀 Quick Comparison
| Control | Primary question | Typical enforcement point | What it cannot guarantee |
|---|---|---|---|
| Permission | May this action be attempted? | Harness policy and downstream authorization | That the action itself is correct or safe in every context |
| Action limit | How much, how often, where, or how broadly? | Policy gate, quotas, scopes, budgets | That the remaining allowed action is desirable |
| Human approval | Should a person explicitly permit this consequential step? | Before the side-effecting operation | That the reviewer sees every hidden failure or future action |
| Cancellation | Should ongoing work stop? | Run controller plus cancellable tools | Undoing side effects that already happened |
01What permissions mean inside an agent harness
A permission is not simply a property attached to a model. In an agent architecture, the model may propose a tool call, but another component should decide whether that call is acceptable for the current user, target, state, and policy.
This is why it helps to separate five responsibilities:
| Model | Interprets the task and may propose the next action. |
|---|---|
| Agent | Coordinates reasoning, tool selection, state transitions, and completion. |
| Harness | Controls runtime behavior: available tools, policy checks, approval gates, budgets, cancellation, state, tracing, and recovery. |
| Tool | Executes a defined operation against a real system. |
| Downstream system | Applies its own authentication, authorization, validation, and transaction rules. |
The exact boundaries vary. In one architecture, the application owns the whole runner. In another, a platform manages the agent loop and state. OpenAI, for example, documents both an Agents SDK, where your application owns the runtime, and a separate managed Agents API. That is a documented architecture distinction, not a universal definition of what every agent harness must look like. OpenAI Agents overview
Engineering principle
The model can recommend an action; the harness should determine whether that action can cross the next trust boundary.
🍊 Tricky concept: access is not authority
A tool appearing in the model's tool list does not have to mean every invocation is allowed. A useful harness can expose a capability while still constraining the target, arguments, amount, time window, identity, or approval state for each actual call.
🛡️ Safety Check
Risk: Treating a tool's mere availability as permission can give an agent broader authority than the task requires.
Control: Evaluate authorization at the tool boundary and enforce it again in the downstream system.
Remaining risk: A correctly authorized action can still be wrong because the input, business state, or model interpretation is wrong.
🎯 Use this when...
Your agent can cross from information retrieval into an external side effect: financial operations, record changes, communications, deployments, deletions, access changes, or other consequential actions.
02Model choice is not authorization
One of the easiest mistakes to make is assuming that a sufficiently capable model can be trusted to decide when an action is allowed. That turns a policy question into a prediction problem.
Consider two questions:
Question A: "Would cancelling this order help solve the customer's problem?"
Question B: "Is this caller authorized to cancel this order under today's policy?"
The first is a reasoning task. The second is an authorization task. The harness should not quietly turn the second into "ask the model what it thinks."
The distinction also matters for prompt injection. Text retrieved from documents, websites, tickets, emails, or tool outputs can contain instructions that try to steer the agent's next step, and even a strong model can be persuaded by them. The stronger boundary is to make authorization independent of the content that is merely being analyzed.
OWASP's excessive-agency guidance specifically recommends implementing authorization in downstream systems and applying the complete-mediation principle, so that requests made through tools are validated against security policy. OWASP LLM06:2025
🟢 Worked example: separate the decision types
The agent reads an order and determines that cancellation is a plausible remedy.
The harness then asks: Is the order owned by the current customer? Is the order still cancellable? Is the action inside the caller's role? Is this a single-order action or a bulk action? Is approval required?
Only after those checks pass does the tool receive an executable request.
Illustrative policy flow — not a vendor SDK:
model_proposes(action) ↓ normalize_and_validate(action) ↓ check_identity_and_scope(action) # identity comes from the app, not the model ↓ check_action_limits(action) ↓ check_approval_policy(action) ↓ execute_only_if_allowed(action)
Engineering principle
Policy should be a gate around the model, not a hope inside the prompt.
🎯 Use this when...
An agent handles untrusted external content, multiple users, sensitive tools, or actions whose authorization can be represented as explicit policy.
03Designing action limits
A permission answers whether an action class is allowed. An action limit answers how far the system may go.
For an agent, "can cancel orders" is usually too broad. Production policy is more useful when it can express constraints such as:
| Limit type | Example | Why it matters |
|---|---|---|
| Capability | Read orders but do not delete records | Reduces unnecessary functionality |
| Resource scope | Only the current customer's order IDs | Limits cross-user access |
| Quantity | One order cancellation per approved request | Prevents accidental bulk impact |
| Value | Refund above a defined business threshold requires review | Controls financial exposure |
| Destination | Only the approved production tenant, account, or recipient | Prevents unintended environments or recipients |
| Time | Approval expires after a short window | Avoids executing a stale decision |
| Concurrency | At most one destructive operation per customer session | Limits cascading side effects |
| Retry budget | Bound automatic retries on side-effecting calls | Prevents a transient failure from becoming repeated actions |
These limits are not meant to be decorative policy. They should be evaluated from trusted request metadata and authoritative system state. For example, the harness should not trust the model to say that an order belongs to the current customer. Customer identity and order ownership should come from authenticated application state or the downstream service.
🍊 Child-friendly analogy
A library card may let you borrow books, but that does not mean you can borrow 500 books, use someone else's card, keep them forever, or remove books from the building. The limit is part of the permission.
A useful policy model is therefore not simply allow = true/false. Conceptually, it resembles:
decision = policy.evaluate(
identity, # from the application, never from the model
action,
target,
current_state, # read from the authoritative system
limits,
approval_state,
time_context,
)
# decision.outcome is one of: ALLOW | REQUIRE_APPROVAL | DENY
# decision.reason is always recorded, so denials can be explained and audited
Illustrative pseudocode: the exact implementation can be simple or elaborate. A small internal assistant might need only an allowlist and one approval gate. A regulated or high-impact system may need identity-bound scopes, transaction limits, operator controls, audit records, and independent downstream authorization.
🛡️ Safety Check
Risk: A read-only-looking workflow accidentally receives a tool identity with write or delete privileges.
Control: Give the tool the minimum downstream authority required and enforce user-specific or task-specific scope in the destination system.
Remaining risk: A compromised or misbehaving agent can still misuse any capability it legitimately receives, so functionality reduction and action limits remain important.
🎯 Use this when...
The cost of a mistake grows with quantity, value, breadth, or repetition. Action limits are especially useful where "one bad call" and "one hundred bad calls" have very different consequences.
04Human approval without approval fatigue
Human approval is most useful when a person is making a genuine authorization decision, not when the system simply asks for a click before every tool call.
The design question is therefore: which actions should pause? A practical classification is:
| Action class | Typical treatment | Example reasoning |
|---|---|---|
| Read-only | Usually automatic | Reading an order status normally does not change external state. Scope still applies, since reads can leak data. |
| Low-impact write | Policy-controlled | May be automatic if the effect is bounded, expected, and reversible. |
| Consequential write | Approval often appropriate | Financial changes, publishing, access changes, or production mutations may need an explicit decision. |
| High-impact or irreversible action | Strong gate or separate workflow | The cost of an incorrect action is high enough that policy should require stronger evidence or human intervention. |
Also decide who approves. A customer confirming their own request in chat is a different control from a support operator approving a refund. Whichever you use, the approver must be an identity the harness can authenticate, and it must not be the model.
OpenAI's Agents SDK documentation describes human review as pausing a run before sensitive tool work: the run records an interruption, returns a resumable state, and your application approves or rejects the pending item and resumes the same run. The same guide advises putting validation next to the tool that creates the side effect, because agent-level guardrails do not run everywhere in a multi-agent workflow. It also recommends failing closed if a review times out or becomes unavailable. Expiry, one-time use, and a re-check of current state are not part of that basic pattern, so they are yours to add. OpenAI: Guardrails and human review
🍊 Child-friendly analogy
A teacher does not need to approve every pencil mark a student makes. The teacher should step in when the student is about to do something that can have a meaningful consequence. The goal is not maximum supervision. The goal is the right supervision at the right moment.
A strong approval request contains enough context to make the decision meaningful. At minimum, consider:
| Actor | Which authenticated user or service identity is requesting the action? |
|---|---|
| Exact action | What will actually happen if approved? |
| Target | Which order, account, environment, recipient, or resource will be affected? |
| Scope and magnitude | How many resources or how much value is affected? |
| Reason | Why does the agent believe this action is required? |
| Evidence | What authoritative information supports the proposed action? |
| Expiry | How long should the approval remain valid? |
Avoid a vague approval message such as "The agent wants to perform a sensitive operation. Approve?" A reviewer cannot meaningfully evaluate an action they cannot see.
🟢 Worked example: a useful approval request
Requested action: Refund <AMOUNT> to the original payment method for order <ORDER_ID>.
Customer: authenticated customer <CUSTOMER_ID>.
Current state: order is pending, payment captured, amount above the review threshold.
Effect: one refund transaction; order will be cancelled afterward.
Why proposed: customer asked to cancel before fulfillment, and the order is still cancellable.
Approver: a support operator with refund authority. Valid for a short window and for this exact action only.
This structure also supports auditing. Later, an operator can answer not just "Was approval given?" but "What exact action was presented, what information was visible, who approved it, and what actually executed?"
🛡️ Safety Check
Risk: People approve repetitive prompts without inspecting them.
Control: Route only genuinely consequential actions to human review, show the exact proposed effect, and make approvals narrow and time-bounded.
Remaining risk: A reviewer can approve a bad action. Human review reduces one class of failure; it does not make the workflow infallible.
🎯 Use this when...
The action is consequential enough that a human decision adds meaningful control, but frequent enough that an explicit approval boundary can still fit the workflow.
05Cancellation: stop the run, stop the action, or undo the effect?
"Cancel" sounds simple until the agent is already doing work.
Suppose the agent has completed three read operations, requested approval for a refund, and then started an external transaction. A user pressing Cancel can mean several different things:
| Cancellation layer | Meaning | Typical limitation |
|---|---|---|
| Stop scheduling | Do not begin another model or tool step. | Already-started work may continue. |
| Interrupt in-flight work | Ask the currently running tool or executor to stop. | Only possible if the execution layer supports safe cancellation. |
| Cancel pending work | Invalidate queued or approved-but-not-started actions, including pending approvals. | Requires every action to check cancellation status before execution. |
| Compensate | Perform a separate recovery action after a side effect already happened. | Compensation is a new action, not time travel, and needs its own authorization. |
This distinction is visible in current agent tooling. OpenAI's Agents API, which is documented as a beta, lets you cancel the active turn to stop current work while keeping the conversation session. OpenAI: Run and continue sessions
MCP provides another useful example. Its Tasks extension defines explicit task cancellation, but describes it as cooperative and eventually consistent: a cancellation request signals intent, the server acknowledges it, and the task can still end in a terminal state other than cancelled if the work finished first. That is an important systems lesson: a cancel signal is not proof that external work stopped. MCP Tasks extension
🍊 Child-friendly analogy
Pressing the stop button on a blender is not the same as putting the fruit back on the kitchen counter. Stop prevents more work, but it cannot undo material that has already been changed.
Engineering principle
Never document "cancel" as if it means "undo." Define exactly which state transition cancellation controls.
For a side-effecting tool, the harness should ideally track the state of each action:
PROPOSED → AUTHORIZED → AWAITING_APPROVAL → APPROVED → RECHECKED → STARTING → RUNNING
│
┌───────────────┬──────────────┬─────────────────┬───────────┘
▼ ▼ ▼ ▼
COMPLETED FAILED UNKNOWN CANCELLED
│ │
▼ ▼
COMPENSATION_REQUIRED RECONCILING
▼
COMPENSATED
Side exits: DENIED (policy) · REJECTED or EXPIRED (approval) · BLOCKED (state changed at re-check)
Cancel: may be requested at any point and shows as CANCELLING until confirmed.
If the action had already started, it may still end as COMPLETED or UNKNOWN.
Illustrative state machine: the actual states will differ by system. The important part is that the harness can distinguish an action that was merely proposed from one that already crossed the external side-effect boundary, and can tell a requested cancellation from a confirmed one.
🛡️ Safety Check
Risk: The UI says "Cancelled" while an external operation is still running or has already completed.
Control: Make cancellation state reflect confirmed runtime state, and distinguish requested cancellation from confirmed cancellation.
Remaining risk: Some downstream systems cannot stop an operation once committed; the recovery path then becomes reconciliation or compensation.
🎯 Use this when...
Tasks may run for a meaningful duration, call external services, consume compute, or create side effects that a user may reasonably need to interrupt.
06Build a durable control loop
The strongest permission system is not a collection of conditionals scattered across tool handlers. It is a repeatable control loop. One practical pattern is:
1. Receive task 2. Establish caller identity and policy context (from the application) 3. Let the agent reason and propose an action 4. Validate the requested tool and arguments 5. Read authoritative state and evaluate policy and action limits 6. Deny, allow, or require approval 7. If approval is required: record the exact action, suspend, wait for an approval event 8. On approval: verify it matches the exact action and has not expired 9. Re-read authoritative state and re-evaluate policy 10. Check cancellation, then persist an intent record BEFORE the side effect 11. Execute with an idempotency key 12. Verify the resulting state against the system of record 13. Record outcome and evidence 14. Continue, stop, reconcile an unknown outcome, or compensate
Steps 7 to 10 deserve special attention. If approval is required, do not keep a bare boolean called approved=true. Associate the decision with the specific action that was reviewed.
A useful action record can contain an action identifier, authenticated actor, tool name, target resource, normalized arguments, a hash of those arguments, a policy version, approval status and approver, approval timestamp, expiry, execution status, and correlation information.
Engineering principle
Approval should authorize an exact action, not a vague future intention.
For example, "approved: refund order" is weaker than "approved: refund <AMOUNT> for order <ORDER_ID>, action hash <HASH>, policy version <POLICY_VERSION>, before <EXPIRY_TIME>." The latter can be checked mechanically before execution.
🟢 Worked example: stale approval
The agent requests approval to refund order A. A reviewer approves it. Before execution, the order changes to "shipped."
A weak implementation sees approved=true and proceeds.
A stronger implementation re-reads authoritative state, sees that the business condition changed, and blocks the action even though approval was previously granted.
This is why approval should not replace authorization. Approval is a human decision at a point in time; authorization is a policy decision against current state.
The same principle applies to cancellation. Before starting an approved action, the executor checks whether the run has been cancelled. For long-running operations, the tool or worker should also expose a cooperative cancellation path where the underlying system permits it. Note that "check, then execute" is itself a race: a cancel can arrive between the two, which is why the unknown-outcome and reconciliation paths still matter.
# Illustrative pseudocode, not a working SDK example.
# Security checks return explicit decisions. They are not `assert` statements,
# which raise generic errors and can be stripped from optimized builds.
def handle_proposal(run, proposal):
action = normalize(proposal) # never trust raw model arguments
state = read_authoritative_state(action.target)
result = policy.evaluate(run.identity, action, state)
if result.outcome == "DENY":
return reject(action, result.reason)
record = store.create_action(action, run.identity, policy.version,
action_hash=hash_action(action))
if result.outcome == "REQUIRE_APPROVAL":
record.mark("AWAITING_APPROVAL")
return suspend(run, record) # resumes on an approval event
return execute(run, record)
def on_approval_event(run, record, approval):
if approval.action_hash != record.action_hash or approval.is_expired() \
or approval.already_used():
return reject(record, "approval does not cover this action")
approval.mark_used()
return execute(run, record)
def execute(run, record):
state = read_authoritative_state(record.target) # re-check right before acting
result = policy.evaluate(run.identity, record.action, state, approved=record.approved)
if result.outcome != "ALLOW":
return block(record, result.reason) # world changed since approval
if run.cancel_requested():
return record.mark("CANCELLED")
record.mark("STARTING") # persisted BEFORE the side effect
try:
outcome = tool.execute(record.action, idempotency_key=record.id)
except DefinitelyNotExecuted as err:
return record.mark("FAILED", err)
except TimeoutError:
record.mark("UNKNOWN")
return reconcile(record) # read back; never blindly retry
if not verify_against_system_of_record(record, outcome):
return record.mark("UNKNOWN")
return record.mark("COMPLETED", outcome)
That verify step matters. A tool returning an HTTP success, database acknowledgement, or generic "accepted" response is not necessarily proof that the business effect occurred as intended.
🛡️ Safety Check
Risk: A previously approved action executes after the world has changed.
Control: Re-authorize against current trusted state immediately before execution, and make each approval single-use and bound to the exact action.
Remaining risk: Race conditions can still exist between authorization and commit, so the downstream system should use its own transaction and concurrency controls where appropriate.
🎯 Use this when...
Actions depend on state that can change between proposal, approval, and execution.
07The complete running example
Let's connect the pieces with the fictional support-operations agent introduced earlier.
🟢 Scenario
A customer says: "My order has not shipped. Please cancel it and refund me."
The agent has four capabilities: (1) read order status, (2) calculate refund eligibility, (3) cancel an eligible order, and (4) submit a refund transaction.
Policy: cancelling a pending order in the customer's own account is allowed after the customer confirms. Refunds above a business threshold need approval from a support operator.
The two actions are linked, so their order is a design decision. Here the harness handles the refund decision first and cancels only afterward. That way a denied refund does not leave an order cancelled with no refund; if the refund is denied, the agent offers alternatives instead.
- Identity is established. The application already knows which authenticated customer is making the request. The model does not choose the customer identity.
- Read-only inspection. The agent retrieves the order status and learns that the order is pending.
- Proposal. Based on the request and current information, the agent proposes a refund and a cancellation.
- Authorization and limits. The harness confirms that the order belongs to the customer context, that cancellation is permitted in the current state, and that the request covers one order and a permitted amount.
- Approval decision. The refund is above the threshold, so the harness records the exact refund action and suspends the run. The pending approval references that action, not "refund customer."
- Human decision. A support operator approves. The system stores the approver, the decision, and the action hash. Rejection ends the refund path.
- Re-check. The harness re-reads the order. If it shipped while waiting, the action is blocked despite the earlier approval.
- Execution. The intent is recorded first, then the tool performs the refund using the appropriate downstream identity, scope, and an idempotency key.
- Verification. The harness reads the resulting transaction state and records evidence of the outcome. Only then does it proceed to the customer-confirmed cancellation, which goes through the same loop.
- Cancellation during execution. If the customer presses cancel while a long-running downstream operation is active, the harness requests cancellation and invalidates anything pending. If the external system cannot stop the operation, the harness reports that accurately and enters reconciliation or compensation instead of showing "Cancelled."
What each part did
Model: interpreted the request, examined information, and proposed actions.
Harness: bound each action to identity, policy, scope, approval, cancellation state, and the execution lifecycle.
Tool: translated the authorized request into an external system operation.
TRUST BOUNDARY MAP User ↓ Application / identity context ↓ Agent model ↓ proposes Harness: policy + limits + approval + cancellation ↓ authorizes Tool adapter ↓ Downstream API / database / service (enforces its own authorization) ↓ Result + verification + trace
This is the main design lesson of the article: permission is a runtime decision about a concrete action, not a general promise that the agent is trustworthy.
🛡️ Safety Check
Risk: The model is allowed to define the target, identity, or authorization context itself.
Control: Derive security-relevant context from trusted application state and validate tool arguments independently.
Remaining risk: The model can still make a poor recommendation inside its allowed scope. Quality evaluation and business validation are still needed.
🎯 Use this when...
You need a practical way to explain agent permissions to engineers, architects, reviewers, or security teams. A running scenario makes the separation of responsibilities easier to reason about than isolated definitions.
08Failure modes and Safety Checks
Permission failures usually appear as ordinary engineering mistakes rather than dramatic security events. The useful question is not only "Can this happen?" but "What does the system do when it happens?"
| Failure | Consequence | Useful correction |
|---|---|---|
| Broad tool identity | A valid tool call reaches more data or write authority than intended. | Reduce downstream privileges and enforce user or task scope. |
| Approval detached from action | An approval can accidentally authorize a different action, or be reused. | Bind approval to a concrete action record, make it single-use, and expire it. |
| No state re-check | A stale decision executes against changed state. | Re-authorize immediately before execution. |
| Cancellation only at UI level | The interface says stopped while backend work continues. | Propagate cancellation into the runtime and tool layer where supported. |
| Retries repeat side effects | A transient error produces duplicate external actions. | Use idempotency where supported, narrow retry rules, and inspect execution state before retrying. |
| Timeout after a write | The outcome is unknown, and a replay may duplicate it. | Record intent first, mark the action UNKNOWN, and reconcile by read-back. |
| Approval fatigue | Reviewers begin approving without examining the action. | Reserve approval for meaningful decision points and present precise evidence. |
| Unverified completion | The agent reports success based on a tool response that is not proof of business completion. | Verify authoritative post-action state before declaring success. |
🍊 Tricky concept: cancellation is itself a controlled action
A cancellation path should not create a second uncontrolled channel that bypasses authorization, tracing, or cleanup. The system should know which run or action is being cancelled, who requested it, and what state the cancellation request reached.
Prompt injection deserves special attention because it can influence an agent's decision without changing the harness policy. For example, a retrieved document could contain text that tries to persuade the agent to call an unrelated administrative tool. The harness should not treat that text as an authority grant. Tool policy, identity, and downstream authorization must remain independent.
No prompt filter, sandbox, approval step, or single validation layer completely eliminates this class of risk. The safer design is defense in depth: limit available capabilities, constrain permissions, validate action arguments, require approval for appropriate high-impact operations, isolate execution where needed, and make consequential activity observable.
🛡️ Safety Check
Risk: Untrusted content causes the agent to request an action outside the user's intended task.
Control: Treat external content as data, keep authorization outside model reasoning, enforce least privilege, and require approval for appropriate side effects.
Remaining risk: The agent may still choose a harmful action that falls within its legitimate authority, so business rules and post-action verification remain necessary.
🎯 Use this when...
You are moving from a demo agent to a production workflow and need to test not only successful paths, but also stale approvals, retries, cancellation races, unauthorized targets, and partial execution.
09Enterprise Rollout
Enterprise adoption is less about adding a single approval screen and more about assigning ownership for the control surface.
- Security ownership: define which controls must be enforced by identity systems, gateway policy, the harness, tools, and downstream applications.
- Application ownership: define what the agent is allowed to do, what requires review, and what evidence proves completion.
- Operations ownership: define who can stop runs, revoke access, disable tools, or activate an emergency deny policy.
- Audit ownership: define retention, traceability, approval evidence, and access to consequential action records.
- Change ownership: version action policies and review changes as carefully as other production control logic.
🟢 Practical rollout sequence
Phase 1: Start with read-only tools and establish identity, traceability, and completion checks.
Phase 2: Add bounded low-risk writes with explicit action limits.
Phase 3: Introduce human approval at clearly defined high-impact boundaries.
Phase 4: Add cancellation propagation, durable action state, recovery, and compensation paths where required.
Phase 5: Test the whole workflow under stale state, retries, partial failures, denied approvals, and malicious external content.
Do not assume that every architecture needs every component. A simple internal agent may not need a full sandbox or multi-stage approval workflow. Conversely, an agent performing production infrastructure changes may need substantially stronger isolation and authorization than a support chatbot. The control surface should be proportional to the consequence of failure.
Engineering principle
Do not copy a security architecture because another agent platform uses it. Start with your own trust boundaries, side effects, identities, and failure costs.
🛡️ Safety Check
Risk: The organization treats vendor defaults as its enterprise authorization model.
Control: Map the agent workflow to enterprise identities, downstream permissions, incident controls, and explicit ownership before production rollout.
Remaining risk: Governance controls can become stale. Policies and access must be reviewed as tools, workflows, and business rules change.
🎯 Use this when...
You are turning an agent prototype into an enterprise service with multiple teams, production identities, incident procedures, compliance requirements, or customer-impacting side effects.
10Common Mistakes
1. "The model has the tool, so it must be allowed."
Why it happens: developers treat tool registration as authorization. Correction: separate capability exposure from per-action authorization and enforce downstream permissions.
2. "Approve this category forever."
Why it happens: broad approvals are convenient. Correction: bind approvals to concrete actions, scoped identities, and expiry conditions.
3. "Cancel means undo."
Why it happens: the UI hides execution state. Correction: distinguish cancellation requests, execution stopping, and compensation.
4. "The approval is valid because the user clicked it."
Why it happens: approval is treated as a final authorization token. Correction: re-check live business state immediately before the side effect.
5. "Retry the tool until it works."
Why it happens: retry logic is copied from read-only integrations. Correction: side-effecting operations need stricter retry semantics, state inspection, and idempotency mechanisms where supported.
6. "Ask for human approval on everything."
Why it happens: humans are seen as the safest generic control. Correction: reserve approval for meaningful decision points and use automatic policy checks for routine low-risk actions.
7. "The tool succeeded, so the task succeeded."
Why it happens: transport success is confused with business success. Correction: define completion criteria and verify authoritative state before reporting completion.
8. "Permissions are only a security-team concern."
Why it happens: authorization is separated from application design. Correction: developers, architects, security, operations, and business owners should agree on the action policy before release.
9. "The agent-level guardrail covers every tool."
Why it happens: checks are attached to the agent and assumed to apply everywhere. Correction: in multi-agent or handoff workflows, agent-level checks may not run for every call, so put validation next to the tool that causes the side effect.
10. "If the reviewer is unavailable, proceed."
Why it happens: timeouts are treated as an availability problem rather than a security decision. Correction: fail closed. An unavailable or timed-out review means the action does not run.
🍊 A useful mental model
When reviewing an agent tool, ask five questions: Who? What? Where? How much? What happens if we need to stop? Those five questions often expose missing controls faster than reading a large tool description.
🛡️ Safety Check
Risk: The team tests only happy-path prompts.
Control: Test denial, stale approvals, reused approvals, changed state, cancellation races, duplicate retries, unauthorized targets, tool outages, and partial side effects.
Remaining risk: Testing samples behavior. Production systems can still encounter conditions the test suite does not cover.
11❓ FAQ
1. Should every agent tool call require human approval?
No. Approval is most useful for meaningful consequential actions, while low-risk operations can often be controlled through explicit policy, narrow permissions, and action limits.
2. Is a human approval the same as authorization?
No. Authorization determines whether an action is permitted under policy against current state, while approval is an explicit human decision that may be required for a particular action at a particular time. An action normally needs both.
3. Can cancellation guarantee that an external action never happens?
No. Cancellation can stop future work or interrupt cooperative operations, but an external side effect that already committed may require separate verification or compensation.
4. Why should the harness re-check state after approval?
Because the world can change between approval and execution, making the previously approved action stale or invalid.
5. Where should permission enforcement ultimately live?
The harness should enforce its own action policy, but the downstream system should also enforce authorization so a model or tool cannot bypass the real security boundary.
12🔗 References & Further Reading
Vendor documentation and draft standards change quickly, so check the linked pages before relying on any detail.
- OpenAI — Guardrails and human review — approval interruptions, resumable state, tool-level validation, and fail-closed review.
- OpenAI — Running agents — further reading on the agent loop and how pauses and resumptions fit into it.
- OpenAI — Agents overview — application-controlled SDK runtimes versus the managed Agents API.
- OpenAI — Run and continue sessions — cancelling an active turn in the beta Agents API.
- OWASP GenAI Security Project — LLM06:2025 Excessive Agency — excessive functionality, permissions, and autonomy; downstream authorization; human approval.
- NIST NCCoE — Software and AI Agent Identity and Authorization (draft concept paper) — proposed demonstration project on identity and authorization for agents.
- Model Context Protocol — Tasks extension — task states and cooperative, eventually consistent cancellation.
Source note
Vendor names, product names, specifications, and API identifiers belong to their respective owners.
13📝 Summary
- An agent can propose an action without being authorized to execute it.
- Permissions should be narrow and enforced at the tool and downstream security boundaries.
- Action limits control scope, quantity, value, destination, time, concurrency, and retries.
- Human approval is most useful at consequential decision points, not as a blanket confirmation screen, and it should fail closed.
- Approval should bind to a concrete action, be single-use, and not survive a meaningful change in authoritative state without another check.
- Cancellation means different things at different layers; stopping a run does not automatically undo external side effects.
- Production-grade harnesses need durable action state, intent records, verification, tracing, controlled retries, and recovery paths.
- No single safeguard makes an agent completely safe; the goal is bounded authority and controlled failure.
Comments
Post a Comment