When an AI system can only answer questions, governance mainly concerns what information it can see and how its outputs are used. When that system can call tools, modify records, send messages, create transactions, or act without another person checking every step, the governance problem changes.
You are no longer governing only an answer. You are governing a permissioned actor inside a larger system.
Imagine an internal document assistant that begins as a search-and-answer service. It can retrieve company policies and explain them to employees. Later, someone gives it two new capabilities: update a service record and send an email to a supplier. The model itself may not have changed at all, yet the risk profile of the deployed AI system has changed substantially.
This is the central governance lesson of this article: permissions, autonomy, and actions must be governed at the level of the complete AI system, not treated as properties that can be delegated to the model alone. The application, tool layer, identity system, downstream authorization, approval process, monitoring, and people who operate the system all become part of the control boundary.
The examples below are teaching scenarios built around a fictional assistant and generic example records. They are not descriptions of any real company's implementation. Framework and standard status notes in this post were checked on October 8, 2026.
01. Why permissions become a governance problem when agents can act
02. Functionality, permissions, and autonomy are three different decisions
03. Build an action risk model before granting capabilities
04. Design a permission envelope, not a vague “agent access” role
05. Make human approval meaningful
06. Test, monitor, and retain evidence
07. Implementation in practice: a fictional document assistant
08. Common governance mistakes
| Decision | What it answers | Governance question | Typical control |
|---|---|---|---|
| Functionality | What operations exist? | Does the agent need this capability at all? | Minimize available tools |
| Permission | Which resources and operations may it use? | What identity, scope, and downstream authorization apply? | Least privilege and complete mediation |
| Autonomy | How freely may the system decide and execute? | Which actions require confirmation, limits, or stopping conditions? | Approval gates, action limits, stop controls |
01
Why permissions become a governance problem when agents can act
A conventional software permission model already asks an important question: what is this identity allowed to do? An AI-enabled system adds another question: how much discretion does the AI have in choosing and combining those allowed operations?
Think about giving a child a school backpack. Functionality is deciding which tools are placed in the backpack. Permission is deciding which rooms the child may enter. Autonomy is deciding whether the child must ask before using a tool or may decide independently. Packing tools nobody needs is a functionality problem. Handing over a key to every room just because one room is needed is a permission problem. Letting the child use a sharp tool in an allowed room without ever checking with an adult is an autonomy problem.
The distinction matters because an agent can turn a relatively harmless capability into a larger chain of consequences. A read operation might expose information. A write operation can change business state. A send operation can create an external commitment. A delete operation may be difficult or impossible to reverse. A sequence of individually modest actions can therefore produce a high-impact outcome.
OWASP's guidance on excessive agency treats excessive functionality, excessive permissions, and excessive autonomy as the typical root causes of harmful agent behavior. It appears as LLM03:2026 Excessive Agency in OWASP's 2026 edition of the Top 10 for LLM Applications (listed on OWASP's site with an August 2026 date) and as LLM06:2025 in the previous 2025 edition. In both editions, the mitigation guidance includes minimizing the tools an agent can call and what those tools can do, reducing downstream privileges, using the user's authorization context where appropriate, requiring approval for high-impact actions, and checking authorization in the downstream systems instead of trusting the model to decide whether an action is permitted. These are security recommendations, not universal legal requirements.
That is also why a model card, benchmark, prompt evaluation, or statement that a model is “safe” does not establish that an agent is safe to operate with a particular business identity. The governance object is the deployed system and its context.
A model should generate or recommend an action; the surrounding system should enforce whether that action is authorized.
Risk: The agent chooses a tool and downstream operation that the business never intended to expose.
Control: Expose only purpose-specific tools, restrict their backend privileges, and enforce authorization outside the model.
Evidence: Approved tool inventory, permission configuration, authorization policy, test results, and action logs.
Remaining risk: A legitimate permission can still be misused because a prompt, retrieved document, user, or connected system may cause an unexpected action.
An AI system can access anything beyond information retrieval, especially when it can modify state, communicate externally, initiate transactions, or perform repeated actions without fresh human confirmation.
02
Functionality, permissions, and autonomy are three different decisions
Governance becomes much clearer when these concepts are separated deliberately.
Functionality is about the operation that exists. For example, “send supplier message” is a function. It answers whether the capability is available to the agent at all.
Permission is about the scope in which that function can operate. An agent might be allowed to send messages only to a defined supplier domain, only from a particular service identity, and only for a particular workflow. That is a permission decision.
Autonomy is about how much independent choice and execution is permitted. A system might draft a message automatically but require a person to approve it. Another system might send a routine notification automatically under tightly bounded conditions. Both have the same underlying function, but different autonomy boundaries.
| Scenario | Function available? | Permission | Autonomy |
|---|---|---|---|
| Read internal policy documents | Yes | Read-only, approved repositories | Usually automatic within scope |
| Update a service record | Yes | Only assigned records and permitted fields | Bounded or approval-based |
| Send an external message | Yes | Restricted recipients, identity, and content scope | Often approval-based |
| Delete a customer record | Yes | Very narrow, privileged operation | Usually human-confirmed |
A common governance mistake is to grant a broad “agent role” and assume that the model will naturally choose to behave within acceptable boundaries. That reverses the control relationship. The safer pattern is to define the permitted operating envelope first and let the model operate inside it.
Risk guidance increasingly recognizes that risk looks different at different levels. The NIST AI RMF Generative AI Profile (NIST AI 600-1, July 2024) describes risks that can sit at the level of an individual model or system, at the application or implementation level for a specific use case, and at the broader ecosystem level. This article reads that as placing most permission governance at the application and operational level, even though model behavior can trigger or amplify the risk.
Version A can search approved documents and produce an answer. Version B uses the same model but can also change a service record. Version C can change the record and send an external message. The model may be identical across all three versions, but the governance controls should not be identical because the system's available actions, blast radius, reversibility, and external consequences are different.
A project discussion starts with “Which model should we use?” but the consequential question is actually “What should this system be allowed to do after the model produces a decision?”
03
Build an action risk model before granting capabilities
There is no universal list saying every AI agent must have the same permission levels. The correct boundary depends on the task, affected people, information involved, downstream systems, and potential consequences.
A practical governance team can still use a repeatable decision method. For every candidate action, assess at least six dimensions:
Impact: What happens if the action is wrong?
Reversibility: Can the outcome be undone safely and completely?
Scope: Does the operation affect one record, one user, a department, or many people?
Externality: Does the action affect a person, customer, supplier, regulator, public channel, or another organization?
Repeatability: Could the agent perform the action repeatedly before someone notices?
Uncertainty: How much interpretation is required before the system can act correctly?
These are governance decision dimensions, not a claimed industry-standard scoring system. An organization can combine them into qualitative tiers, a numerical score, or a decision matrix appropriate to its risk-management method.
Imagine four chores. Reading a shopping list is easy to undo. Rearranging the living room is more significant but reversible. Ordering an expensive item is more serious. Throwing away something important may be irreversible. The fact that one child can perform all four chores does not mean the child should be allowed to perform all four without different checks.
The result can be a simple action taxonomy such as read, draft, create, modify, send, approve, delete, or execute privileged operation. The labels themselves are not important. What matters is that the organization can explain why different operations receive different controls.
| Illustrative action tier | Example action | Typical treatment | Why |
|---|---|---|---|
| Low | Read approved knowledge | Automatic within defined scope | Limited direct side effects (confidentiality and privacy risks still apply) |
| Moderate | Create or update an internal record | Bounded scope, validation, logging, possible approval | Business state changes |
| High | External communication with commitment or consequence | Strong validation, restricted scope, human confirmation where appropriate | Effects leave the system boundary |
| Very high | Irreversible, privileged, or consequential operation | Narrow identity, explicit approval, strong evidence, emergency stop path | Large or difficult-to-reverse harm potential |
Risk: Every tool receives the same “agent access” level because creating separate policies appears inconvenient.
Control: Evaluate every action by impact, scope, reversibility, externality, repeatability, and uncertainty before granting access.
Evidence: Action inventory, impact rationale, approved permission tier, reviewer, and test cases showing the boundary works.
Remaining risk: Risk classification can become stale when a workflow, connected system, or agent capability changes.
You need to decide whether an agent should be allowed to perform an action automatically, perform it only under conditions, or require a human decision first.
04
Design a permission envelope, not a vague “agent access” role
A useful governance artifact is a permission envelope: a documented statement of what an agent may do, where it may do it, under whose authority, under which conditions, and what evidence must be recorded.
A permission envelope is an original teaching construct in this article rather than a named requirement from a particular standard. It can be implemented using existing identity, authorization, API gateway, workflow, policy, and monitoring mechanisms.
Permission Envelope
─────────────────────────────────────────────
Agent identity: <AGENT_ID>
Business purpose: Service-case assistance
Allowed tools: retrieve_policy, read_case, update_case_status,
draft_supplier_message, send_supplier_message
Allowed resources: Assigned service cases only
Allowed operations: Read; status field only; drafts; approved sends
Identity context: Authenticated user + service identity
Approval required: Every send_supplier_message call;
status changes to a closing value
Denied by default: Deletion, bulk updates, any tool not listed
Rate limit: Set by the process owner
Time boundary: Session or task expiry
Logging: Request, authorization, decision, result
Stop condition: Policy violation, repeated failure, manual stop
Reviewer: Process owner / risk ownerIllustrative only: this record is a governance teaching artifact, not a legally sufficient access-control template.
Notice what is missing from this example: a statement such as “the agent is trusted.” Trust is not a technical authorization boundary. The useful questions are concrete: which tool, which resource, which operation, under which identity, for how long, with what constraints, and subject to which approval?
Least privilege is especially important when the model is connected to a downstream system. OWASP's excessive-agency guidance gives the example of a tool that only needs to read data but connects with an identity that can also update, insert, and delete. The governance lesson is broader than databases: the downstream identity should have only the authority needed for the approved operation.
The permission boundary should also exist independently of the model's own reasoning. A prompt should not be the mechanism that grants authority. A memory record should not silently expand authority. A retrieved document should not redefine security policy. A model-generated statement such as “the user is authorized” is a claim to be checked, not evidence of authorization.
Authentication identifies an actor; authorization constrains what that actor may do; model reasoning should never be treated as a substitute for either.
The same principle matters in multi-agent designs. Agent A should not automatically inherit every capability available to Agent B simply because the two agents exchange messages. A handoff should specify what task context is transferred, what authority is transferred, and what authority is not transferred. In many designs, it is safer for the receiving agent to authenticate and authorize its own downstream action rather than inheriting an unlimited capability from its peer. OWASP's 2026 guidance makes a related point for delegated workflows: preserve the original user's authorization scope across chained calls instead of relying only on the calling agent's identity.
Suppose an assistant only needs to update the status of an assigned support case. A broad generic database tool may technically make the task easy, but it creates unnecessary capabilities. A narrower application operation such as “update assigned case status” makes the allowed business operation explicit and allows the downstream service to validate ownership, permitted values, and identity before changing state.
The architecture currently exposes broad database, shell, browser, mailbox, file-system, or generic API capabilities when the business process needs only one narrow operation.
05
Make human approval meaningful
“Human in the loop” sounds reassuring, but a human button does not automatically become effective oversight.
A meaningful approval should give the reviewer enough information and authority to make an informed decision. Depending on the action, that can include the exact operation, target, affected records, important input data, proposed result, business reason, relevant uncertainty, and consequence of approval.
Imagine a person standing next to a self-driving forklift and being told to press “OK” every time it starts moving. If the person cannot see where it is going, cannot understand why it is moving, and cannot stop it, the presence of a human does not create meaningful oversight. AI approvals have the same problem.
A stronger approval gate can follow this pattern:
- Identify the action. State exactly what will happen.
- Identify the target. Show the record, recipient, transaction, or resource affected.
- Explain the consequence. Show what changes if the reviewer approves.
- Make the decision independent. The reviewer should approve or reject the action rather than merely confirm that the model produced it.
- Enforce the decision downstream. The approval should be checked by the system that performs the consequential operation.
- Record evidence. Retain who approved what, when, under which policy, and what eventually happened.
The OECD AI Principles, as updated in May 2024, emphasize human agency and oversight, transparency, accountability, and traceability. They also call for mechanisms that let people override, repair, or safely decommission AI systems that cause undue harm or behave in undesired ways (see OECD's announcement of the update). Those principles do not prescribe one universal approval screen. The governance task is to translate the applicable principle into a control that actually works in the specific workflow.
Approval record ───────────────────────────────────────────── Request ID: <REQUEST_ID> Requested action: Send supplier message Target: <SUPPLIER_REFERENCE> Proposed content: <MESSAGE_PREVIEW> Reason: Case resolution workflow Policy trigger: External communication Decision: Approved / Rejected Reviewer: <REVIEWER_ID> Decision time: <TIMESTAMP> Authorization checked: Yes / No Execution result: Success / Failure Evidence location: <AUDIT_RECORD_REFERENCE>
Illustrative only: field names, retention requirements, and approval rules must come from the organization's actual governance and control environment.
Risk: A reviewer approves actions reflexively without seeing the actual consequence.
Control: Present the exact action and target, provide enough context to evaluate it, and allow explicit rejection or escalation.
Evidence: Approval record linked to the exact action request and resulting downstream transaction.
Remaining risk: Human reviewers can still make mistakes, approve too quickly, or lack sufficient domain context.
An AI action can create a consequential external effect, financial commitment, sensitive disclosure, destructive change, or other outcome where an informed person should retain decision authority.
06
Test, monitor, and retain evidence
A permission policy is not proven merely because the configuration exists. Governance needs evidence that the control behaves as intended under normal use, edge cases, foreseeable misuse, and change.
A useful test plan should include at least five types of questions:
Allowed action: Can the agent perform an approved operation with valid inputs?
Denied action: Does the system reject an operation outside the approved scope?
Manipulated action: What happens when user text, retrieved content, tool output, or another agent attempts to induce an unauthorized operation?
Repeated action: What happens when the same operation is attempted many times, or continues after an error?
Changed environment: Does the control still work after a tool, identity, API, prompt, model, workflow, or data source changes?
This is especially important because AI systems are socio-technical. NIST's AI RMF describes risk as something that can arise from the way the technology, its users, its operators, and its social setting interact. The Measure section of the NIST AI RMF Playbook lists suggested actions for testing a system before deployment and monitoring it in production.
For agent permissions, useful evidence may include tool inventory, identity configuration, authorization rules, negative test results, approval records, execution logs, incident records, monitoring alerts, and change-control history. The objective is not to collect every possible log. It is to retain enough evidence to reconstruct a consequential decision and assess whether the control operated as intended.
The fictional service assistant is authorized to update the status field of a case assigned to the current user. A test asks it to update a case owned by another team. The correct result is not merely that the model refuses in natural language. The downstream service should reject the unauthorized request even if the model attempts it. The test evidence should show the attempted operation, authorization decision, resulting response, and preserved audit record.
That distinction is central to secure agent governance: test the boundary where authority is enforced, not only the boundary where the model is instructed.
Monitoring also needs to look for behavioral signals such as unusual action frequency, repeated authorization failures, attempts to access resources outside normal scope, unusual recipient patterns, unexpected tool sequences, or repeated execution after an error. Rate limits and monitoring can reduce impact, but they are not substitutes for proper authorization: OWASP classes logging, monitoring, and rate limiting as damage-limiting measures that do not prevent excessive agency by themselves, and puts the primary emphasis on minimizing capabilities and enforcing authorization.
Risk: The organization tests only happy-path prompts and assumes the permission boundary works.
Control: Test allowed, denied, manipulated, repeated, and changed-environment cases at the actual authorization boundary.
Evidence: Test cases, expected decisions, actual results, control version, date, reviewer, and remediation records.
Remaining risk: Testing provides evidence for the tested conditions; it does not prove that every future behavior is safe.
07
Implementation in practice: a fictional document assistant
Consider a fictional internal assistant called Atlas Desk. This name and scenario are invented for teaching purposes.
In its first version, Atlas Desk retrieves approved internal documents and answers employee questions. It has no write access to operational systems. Governance focuses mainly on information access, source boundaries, accuracy evaluation, user transparency, and monitoring.
The business later requests two new features:
Feature A: update the status of a service case after reviewing the relevant documents.
Feature B: send a message to an external supplier after a case reaches a defined condition.
A governance review should not simply ask, “Can the model perform these tasks?” It should ask:
Who owns the business decision to permit each action?
Which records can be changed?
Which fields can be changed?
Whose identity authorizes the change?
Does the agent need to send messages, or only draft them?
What happens when a document contains instructions that conflict with the business process?
How is a prohibited operation blocked?
How can the organization stop the capability quickly if unexpected behavior appears?
| Control decision | Fictional decision for Atlas Desk | Evidence |
|---|---|---|
| Purpose | Assist with internal service-case resolution | Approved use-case record |
| Read scope | Approved documents and assigned case context | Repository policy and access tests |
| Update scope | Status field only, for authorized cases | Authorization tests and execution logs |
| Supplier communication | Draft automatically; send only after a named person approves that specific message | Approval record and message audit record |
| Emergency control | Disable the affected tool capability without shutting down all information access | Runbook and periodic stop-control test |
This design illustrates a useful idea: not every capability needs the same autonomy level. Reading an approved document may be allowed automatically. Updating a specific business field may require validation and strict authorization. Sending an external communication may require an explicit approval gate. Deleting a record may be prohibited entirely or reserved for a highly controlled workflow.
Suppose a retrieved document contains text telling the assistant to bypass approval and send a message immediately. The correct governance response is not to rely on the model “knowing” that the document is untrusted. The tool execution path should enforce the approval requirement independently. The retrieved text may influence reasoning, but it should not grant authority that the workflow did not authorize.
This is one reason the governance boundary should include the surrounding application, tool layer, identity layer, and downstream system. The model is one component of the decision chain, not the complete authorization system.
Risk: New tool capabilities are added to an existing assistant without reopening the original governance decision.
Control: Treat material capability or autonomy changes as change events that trigger reassessment, testing, and approval.
Evidence: Version history, change ticket, updated permission envelope, regression test results, and re-approval.
Remaining risk: Some interactions among new tools, data, agents, and workflows may not become visible until operation.
An existing AI assistant gains a new connector, tool, memory capability, downstream identity, autonomous loop, or external action. Treat the change as a governance event rather than merely a feature release.
08
Common governance mistakes
| Mistake | Why it happens | Better correction |
|---|---|---|
| “The prompt says the agent must not delete anything.” | Instruction is mistaken for authorization. | Do not expose a deletion capability at all, and enforce the restriction at the downstream authorization boundary. |
| “Give the agent the same service account as the application.” | Shared identities are operationally convenient. | Separate or narrowly scope identities and permissions so the AI capability receives only the authority it needs. |
| “A human approves every action, so governance is solved.” | Human presence is easier to demonstrate than meaningful oversight. | Design informative approval, decision authority, downstream enforcement, and evidence. |
| “A tool has been tested once.” | Testing is treated as a launch activity instead of an operating discipline. | Retest after material changes, incidents, permission changes, new integrations, and relevant model or workflow changes. |
| “Memory can remember that this user is trusted.” | Memory is confused with authorization state. | Keep authorization authoritative in the identity and policy layer. Memory may support context; it should not silently grant privilege. |
A related mistake is evaluating only the model while ignoring the full system. A model may be acceptable in an isolated test environment, yet become risky when connected to a broad identity, unrestricted tools, persistent memory, unattended execution, or sensitive downstream systems.
Another mistake is assuming that a governance framework itself determines the correct permission. Frameworks and standards can provide principles, processes, or management-system expectations; the organization still has to make a context-specific risk decision. ISO/IEC 42001, for example, is a management-system standard that organizes AI governance as a plan-do-check-act cycle across an organization. It is not a technical permission matrix for agents.
Similarly, the NIST AI RMF is voluntary. As of October 8, 2026, NIST's AI RMF page reports that a revision of AI RMF 1.0 is under way, tied to the White House AI Action Plan, and that the Playbook will follow once it is done. Check which version and which organizational policy actually apply rather than treating any framework name as a universal legal requirement.
A governance review is becoming a checklist exercise. Ask instead: “What exact action could happen, who authorizes it, what blocks the wrong action, what proves the control worked, and what remains possible if the control fails?”
09
❓ FAQ
No. Approval should be proportionate to the action and its consequences. Low-impact, reversible actions may be appropriate for bounded automation, while high-impact, external, destructive, or difficult-to-reverse actions may require stronger confirmation or additional controls.
The model should not be the authoritative authorization mechanism. Identity and authorization controls should enforce whether the requested operation is permitted, while the model can help determine what operation it wants to request.
No. Read-only access can still create confidentiality, privacy, inference, or downstream decision risks, especially when sensitive information is exposed or combined with other capabilities. Read-only should therefore be treated as lower action risk in a particular context, not as risk-free.
A new tool can change the system's functionality, permissions, attack surface, affected people, and potential consequences. Treat material tool additions as a change event: reassess the use case and risk, update the permission envelope, test the new boundary, confirm ownership, and retain evidence of the decision.
There is no single universal evidence item. A strong evidence set connects the approved purpose and permission scope to the actual enforced authorization rule, test results for allowed and denied actions, relevant approval records, monitoring data, and change history. The stronger the consequence, the stronger the evidence should generally be.
10
🔗 References & Further Reading
Framework and project names remain the property of their respective owners. Status notes below were checked on October 8, 2026.
NIST AI Risk Management Framework: official source NIST's official AI RMF page. States that the framework is for voluntary use and, as of October 8, 2026, that a revision of AI RMF 1.0 is under way, tied to the White House AI Action Plan.
NIST AI RMF 1.0 (NIST AI 100-1, January 2023): official source The framework itself: the four Core functions Govern, Map, Measure, and Manage, and the description of AI systems as socio-technical.
NIST AI RMF Playbook: Measure: official source Suggested actions for pre-deployment testing and production monitoring. The page notes the Playbook will be updated after AI RMF 1.0 is revised.
NIST Generative AI Profile (NIST AI 600-1, July 2024): official source Describes generative-AI risks at the model or system, application, and ecosystem levels, with suggested actions across the AI RMF functions.
OWASP Top 10 for LLM Applications 2026: LLM03:2026 Excessive Agency: official source Current edition, listed on OWASP's site with an August 2026 date. Excessive functionality, permissions, and autonomy, with mitigations such as minimizing tools, least privilege, user-context execution, approval for high-impact actions, and complete mediation.
OWASP LLM06:2025 Excessive Agency (previous edition): official source The same risk under its 2025 number. Kept for readers whose documentation still cites the 2025 edition.
OWASP Top 10 for Agentic Applications for 2026: official source Published December 9, 2025. Security risks specific to agents that plan, use tools, and act autonomously.
OWASP Agent Control Standard (ACS): official source Listed on OWASP's site with a September 1, 2026 date. Describes agents that are inspectable, traceable, and instrumentable, with controls enforced at runtime. An OWASP project resource, not a universally binding requirement.
OECD AI Principles: official source The OECD's overview of the first intergovernmental AI standard, including human agency and oversight, transparency, robustness and safety, accountability, and traceability.
OECD announcement of the May 2024 update: official source Confirms the 2024 revision, including mechanisms to override, repair, or safely decommission AI systems.
OECD Due Diligence Guidance for Responsible AI (published February 19, 2026): official source Practical guidance for enterprises on applying OECD responsible-business-conduct standards and the OECD AI Principles so they can address adverse impacts proactively.
ISO/IEC 42001: official source ISO's overview of the AI management-system standard. Cited here only to distinguish an organizational management system from a technical permission model for agents.
11
📝 Summary
- An agent's permission is a system governance decision, not merely a model configuration.
- Separate functionality, permission, and autonomy. Each creates a different governance question.
- Minimize capabilities before trying to supervise them. A capability that does not need to exist is easier to govern than one that merely has a warning around it.
- Enforce authorization outside the model. The model may request an action; the surrounding system should decide whether that action is permitted.
- Make human approval meaningful. Show the real action, target, consequence, and decision context.
- Test denied actions as seriously as allowed actions. A permission boundary is valuable because it blocks what should not happen.
- Treat new tools, identities, autonomy, and external actions as governance change events.
- Keep evidence. A professional governance decision should be explainable after the system has been deployed.
The most useful mental model is simple: give the agent enough authority to accomplish its approved job, but not enough authority to redefine the job. Governance succeeds when policy becomes an enforceable boundary, the boundary can be tested, evidence can be retained, and someone clearly owns the remaining risk.
Build useful agents. Give them bounded authority. Keep consequential decisions reviewable.
Comments
Post a Comment