Multi-Agent Architecture - OIC Projects Capabilities

Calculating read time…

Let's say your company receives 500 new employee joining requests every Monday morning. Each request needs: HR to create the employee profile in Oracle HCM, Finance to set up the payroll account in Fusion Payroll, IT to provision system access via an ITSM tool, and Facilities to assign a desk and equipment via a facilities system.

🎯

What is Multi-Agent?

Team of AI agents

🏗️

OIC Gen3 Project

The container

👑

Orchestrator Agent

The team manager

🤝

Sub-Agents

The specialists

🔧

Shared Tools

Shared capabilities

🛡️

Security

Enterprise-grade

📊

Observability

See everything

🎯 Section 1: The hook — a story every developer will relate to

Imagine instead of one AI agent trying to do all four jobs itself — running from HR to Finance to IT to Facilities — you have a team of four specialist AI agents, each an expert in their domain, all coordinated by a smart manager agent who gives each specialist the right task at the right time, waits for results, and hands the next task to the next specialist.

✅ That is Multi-Agent Architecture in OIC Gen3. A coordinated team of specialised AI agents, each doing one job brilliantly, working together under the direction of an Orchestrator — all built and deployed as a single OIC Gen3 Project. The whole onboarding workflow completes in minutes, with full audit trail, human approval where needed, and zero manual coordination.

This article explains exactly how to build that — from the very beginning, in plain language, with real Oracle examples throughout.

🏢 Section 2: The business problem — why one agent is not enough

When you build a single AI agent to handle a complex enterprise workflow, it quickly runs into three hard limits that no amount of clever prompting can fix:

⚠️ Problem 1 — Context window explosion

A single agent handling HR + Finance + IT + Facilities needs all four domains' tools, policies, data, and rules loaded into its context window simultaneously. The context fills up fast. The agent starts forgetting earlier decisions. Errors multiply.

⚠️ Problem 2 — Domain confusion

When one agent tries to be an HR expert and a Finance expert and an IT expert, it becomes mediocre at all three. Rules from one domain bleed into another. "Approve payroll" gets confused with "approve system access." Reasoning quality degrades badly.

⚠️ Problem 3 — No parallelism

A single agent works sequentially — do HR, then Finance, then IT, then Facilities. But HR and IT provisioning can run at the same time. A team of agents can work in parallel. What takes 40 minutes sequentially takes 12 minutes with parallel specialised agents.

💡 The solution: Multi-Agent Architecture solves all three problems. Each agent is a focused specialist with its own context window, its own domain expertise, its own tools. The Orchestrator coordinates them efficiently — running tasks in parallel where possible, sequentially where required. Together they handle complexity that no single agent ever could.

Scenario	Single agent approach	Multi-agent approach
Employee onboarding	One overloaded agent handles all 4 domains	HR Agent + Finance Agent + IT Agent + Facilities Agent — coordinated
Supplier onboarding	One agent tries to validate, create, notify — context bloat	Validation Agent + Finance Agent + Comms Agent in parallel
Month-end close	One agent runs all reconciliation tasks sequentially — hours	GL Agent + AP Agent + AR Agent run reconciliations in parallel — minutes
Customer order fulfilment	One agent handles credit check + inventory + logistics — confused	Credit Agent + Inventory Agent + Logistics Agent — each a domain expert

💡 Section 3: Why Multi-Agent Architecture matters in OIC Gen3

OIC Gen3 is the first version of Oracle Integration Cloud that natively supports Multi-Agent Architecture as a first-class design pattern. It is not a hack or a workaround — it is built into the platform through a feature called OIC Gen3 Projects.

💡 Kid-friendly explanation: Imagine you are a teacher organising a school science fair. You could do everything yourself — set up each table, guide each student, judge each project, write each certificate. That would take forever and you'd make mistakes. Instead, you assign a Project Manager (Orchestrator Agent) who coordinates a team: one person sets up tables (Sub-Agent 1), one guides students (Sub-Agent 2), one judges projects (Sub-Agent 3), one writes certificates (Sub-Agent 4). Everyone does their job in parallel. The science fair runs in 2 hours instead of 2 days. That is Multi-Agent Architecture. In OIC Gen3, the entire science fair — the project manager, the team members, and all their equipment — lives inside one thing called an OIC Gen3 Project.

🏛️ Oracle-native

The entire multi-agent system — Orchestrator, Sub-Agents, shared tools, prompt templates, memory — is managed inside OCI tenancy. No external orchestration frameworks needed.

♻️ Reuse existing flows

Every OIC integration your team has ever built can become a Tool for one of your agents. Your existing OIC investment is not replaced — it becomes the action layer of your multi-agent system.

⚡ True parallelism

OIC Gen3 Projects can run Sub-Agents in parallel. HR provisioning and IT access setup happen at the same time, not one after the other. Real-world workflows are 3–5× faster.

🔒 Enterprise guardrails

OCI IAM, Vault, audit trails, human approval gates — all enforced per-agent. Each specialist agent has its own security scope. An HR Agent cannot accidentally call a Finance tool.

🏗️ Section 4: What is an OIC Gen3 Project? — the container for everything

Before we talk about agents, we need to understand the box that holds them all. In OIC Gen3, a Project is a unified workspace that groups all related integration artifacts — agents, tools, prompt templates, connections, and schedules — into one manageable, deployable, version-controlled unit.

💡 Kid-friendly analogy: Think of an OIC Gen3 Project like a LEGO box with a specific theme — say, a LEGO City Fire Station set. The box contains everything that belongs to that set: fire trucks (Orchestrator + Sub-Agents), the fire station building (shared tools and connections), instruction booklets (prompt templates), and the pieces that connect them all. You build it as one complete set, not as random scattered pieces. When you're done, you can pick up the whole box and move it to another city (environment). That is an OIC Gen3 Project.

📦 What lives inside an OIC Gen3 Project

📦 OIC Gen3 Project — complete contents

👑

Orchestrator Agent

The team manager. Receives the top-level goal, breaks it into sub-tasks, delegates to specialists.

🤝

Sub-Agents (N)

Domain specialists. Each has its own prompt template, tool set, and scope. Called by the Orchestrator.

🔧

Tools (shared + private)

OIC integrations registered as tools. Some shared across agents, some private to one agent.

📝

Prompt Templates

One per agent. Defines each agent's identity, rules, reasoning style, and output format.

🔗

Connections

OIC adapter connections (Fusion, REST, DB, Teams) shared across all agents in the project.

⏰

Schedules & triggers

Scheduled or event-based triggers that kick off the Orchestrator. REST triggers for human-initiated workflows.

🎯 Why Projects matter for Multi-Agent

✅ Single deployable unit: all agents, tools, templates, and connections deploy together as one Project. No risk of Orchestrator being deployed without its Sub-Agents.

✅ Shared connections: one Fusion adapter connection defined at Project level — all 5 agents inside the project reuse it. No duplicate connection management.

✅ Isolated security scope: each agent within the Project has its own OCI IAM role. The HR Agent cannot call the Finance tools even though they are in the same Project.

✅ Version as one: when the Onboarding Project goes from v1.0 to v2.0, all agents, templates, and tools version together. Rollback means rolling back the whole team, not just one member.

👑 Section 5: The Orchestrator Agent — the team manager

The Orchestrator Agent is the most senior agent in the project. It receives the high-level business goal — "onboard employee John Smith" — and is responsible for breaking it down into sub-tasks, deciding which specialist agent handles which task, whether tasks run in parallel or sequence, and assembling all the results into a final outcome.

💡 Kid-friendly analogy: The Orchestrator is like the head chef in a restaurant kitchen. When an order arrives — "Table 7 wants a full Italian meal" — the head chef does not cook everything. They read the order, break it into tasks ("pasta, salad, bread, dessert"), give each task to a specialist cook (Sub-Agent), coordinate who starts when, monitor progress, and when everything is ready, plate it all together for the customer. The head chef never touches the pasta directly. They manage the team that does.

👑 What the Orchestrator does — step by step

👑 Orchestrator execution flow — employee onboarding

📥 Receive goal: "Onboard John Smith as Senior Engineer in BU=IN_TECH, start date=July 1, role=SRE, salary_band=L4."

⬇️

🧠 Analyse and plan: "This requires: (1) HR profile creation in Fusion HCM, (2) Payroll account setup in Fusion Payroll, (3) IT system access provisioning, (4) Facilities desk assignment. Steps 1 and 3 can run in parallel. Step 2 requires step 1 to complete first. Step 4 can run with step 2."

⬇️

⚡ Delegate (parallel batch 1): call HR-Agent with employee details → call IT-Agent with role and system access requirements → run both simultaneously.

⬇️

👁️ Collect results batch 1: HR-Agent returns: employee_id=EMP-00823. IT-Agent returns: access_ticket=INC-9934, systems provisioned=5.

⬇️

⚡ Delegate (parallel batch 2): call Finance-Agent with employee_id=EMP-00823 and salary_band=L4 → call Facilities-Agent with employee_id and start_date → run both simultaneously.

⬇️

✅ Assemble and deliver: all 4 specialists complete. Orchestrator assembles full onboarding summary: employee profile created, payroll set up, 5 systems provisioned, Desk D-204 assigned. Sends confirmation to HR Manager via Teams. Total time: 4 minutes 18 seconds.

⚙️ Orchestrator configuration — key differences from a regular agent

🔧 Tools = Sub-Agents

In the Orchestrator's tool catalogue, the "tools" are not just OIC integrations — they also include Sub-Agents themselves. Calling HR-Agent is a tool call from the Orchestrator's perspective. The Sub-Agent appears as a callable REST endpoint.

📋 Higher max_iterations

The Orchestrator manages a more complex workflow than any individual Sub-Agent. Typical setting: max_iterations=40–60. Each Sub-Agent call counts as one iteration from the Orchestrator's perspective.

🌡️ Same low temperature

temperature=0.1 for the Orchestrator. Coordination decisions must be deterministic — wrong delegation order causes data dependency errors. This is not the place for creativity.

📝 High-level prompt template

The Orchestrator's prompt template defines coordination rules, not domain expertise. It knows who to call for what, when to run tasks in parallel vs series, and how to assemble final results from multiple agent responses.

📝 Sample Orchestrator prompt template

## IDENTITY
You are OnboardOrchestrator, the coordination agent for InfraGroup India's
employee onboarding workflow. You do NOT perform domain-specific tasks.
You delegate to specialist agents and assemble their results.
Session: {{session_id}} | Initiated by: {{user_name}} | Date: {{current_date}}

## YOUR AVAILABLE SPECIALISTS
HR-Agent: Creates employee profiles in Oracle Fusion HCM. Call first for any new hire.
Finance-Agent: Sets up payroll in Fusion Payroll. Requires employee_id from HR-Agent.
IT-Agent: Provisions system access. Can run in parallel with HR-Agent.
Facilities-Agent: Assigns desk and equipment. Requires employee_id from HR-Agent.

## COORDINATION RULES
RULE 1: HR-Agent and IT-Agent can always run in PARALLEL on step 1.
RULE 2: Finance-Agent and Facilities-Agent require employee_id — run AFTER HR-Agent completes.
RULE 3: Finance-Agent and Facilities-Agent can run in PARALLEL once employee_id is known.
RULE 4: If any specialist returns an error, pause and notify {{manager_email}} immediately.
RULE 5: NEVER attempt to perform HR, Finance, IT, or Facilities tasks yourself. Always delegate.

## TASK IS COMPLETE WHEN:
All 4 specialists have returned success responses and the onboarding summary
has been sent to {{manager_email}} via the send_onboarding_summary tool.

## OUTPUT FORMAT
ONBOARDING COMPLETE: [Employee Name] | Employee ID: [ID]
HR STATUS: [result] | PAYROLL STATUS: [result]
IT STATUS: [result] | FACILITIES STATUS: [result]
TOTAL TIME: [duration] | NEXT STEPS: [any manual actions required]

🤝 Section 6: Sub-Agents — the domain specialists

A Sub-Agent is a focused, domain-expert AI agent that receives a specific, scoped task from the Orchestrator and executes it to completion. Unlike the Orchestrator, which knows how to coordinate, each Sub-Agent knows how to do. It has deep domain knowledge, all the tools for its area, and a very precise prompt template that makes it brilliant at exactly one job.

💡 Kid-friendly analogy: Sub-Agents are like specialist doctors in a hospital. The head consultant (Orchestrator) receives a patient with multiple issues. They don't treat everything themselves — they refer the patient to a cardiologist for the heart issue, a neurologist for the headache, and a physiotherapist for the knee. Each specialist is the world's best at their specific area. They receive the referral, do their work, and report back. That is exactly how Sub-Agents work.

🤝 The four Sub-Agents — InfraGroup onboarding example

👤 HR-Agent

Domain: Oracle Fusion HCM
Input: Employee details from Orchestrator
Job: Create employee profile, assign role and BU, set start date, generate employee ID
Output: employee_id, profile_url, assignment_id

Tools:
create_hcm_employee()
assign_hcm_role()
get_hcm_lookup()
validate_employee_data()

💰 Finance-Agent

Domain: Oracle Fusion Payroll
Input: employee_id + salary_band from Orchestrator
Job: Set up payroll element, assign salary grade, configure bank account, run payroll simulation
Output: payroll_id, effective_date, net_salary_estimate

Tools:
create_payroll_element()
assign_salary_grade()
get_payroll_lookup()
validate_payroll_config()

💻 IT-Agent

Domain: ITSM + Active Directory
Input: role + start_date from Orchestrator
Job: Create AD account, assign access groups based on role, provision SaaS applications, generate access credentials email
Output: ad_account, systems_provisioned[], ticket_id

Tools:
create_ad_account()
assign_access_groups()
provision_saas_apps()
create_itsm_ticket()

🏢 Facilities-Agent

Domain: Facilities Management System
Input: employee_id + start_date + location from Orchestrator
Job: Find available desk in correct floor, assign equipment kit, create building access card request
Output: desk_id, equipment_kit_id, access_card_request_no

Tools:
find_available_desk()
assign_equipment_kit()
request_access_card()
send_facilities_email()

⚙️ Sub-Agent design principles

🎯 Principle 1 — One domain, maximum depth

Each Sub-Agent owns exactly one business domain. HR-Agent knows everything about Oracle HCM. Finance-Agent knows everything about Fusion Payroll. This depth is what makes them more reliable than a generalist agent. A specialist always outperforms a generalist in their area.

📨 Principle 2 — Accept structured input, return structured output

Sub-Agents must accept a clean JSON input from the Orchestrator and return a clean JSON output. The Orchestrator cannot process free-form text from a Sub-Agent — it needs structured data to make coordination decisions. Define the input/output contract before building the Sub-Agent.

🔒 Principle 3 — Narrow IAM scope

Each Sub-Agent gets an OCI IAM role that grants access only to its domain's tools. The IT-Agent cannot call Fusion Finance APIs. The Finance-Agent cannot call the facilities system. This is not optional — it is the security contract of the multi-agent architecture.

⏱️ Principle 4 — Sub-Agents must be fast

The Orchestrator waits for Sub-Agent results before proceeding. A slow Sub-Agent blocks the whole workflow. Set aggressive timeouts (sub-agent session_timeout=3 minutes max) and design Sub-Agent tasks to be atomic — one clear goal that completes quickly, not multi-step complex workflows. Reserve complexity for the Orchestrator.

🔧 Section 7: Shared tools & connections — the team's shared equipment

Inside an OIC Project, some tools are private to one agent (only the HR-Agent can create HCM employees). Other tools are shared across multiple agents (every agent needs to write to the audit log, send Teams notifications, or search the knowledge base). OIC Gen3 Projects manage this elegantly.

💡 Kid-friendly analogy: In a school library, each teacher has their own classroom materials (private tools). But the library itself — books, computers, the printer — is shared by all teachers (shared tools). The school administration manages the library centrally so every teacher can access it without each teacher having their own copy of every book. That is how shared tools work in an OIC Gen3 Project.

Tool	Type	Who uses it	OIC backing	Classification
`send_teams_notification`	🟡 Shared	All agents	OIC → Teams Webhook	WRITE-REVERSIBLE
`write_audit_log`	🟡 Shared	All agents	OIC → OCI Logging	READ
`search_policy_kb`	🟡 Shared	All agents	OIC → OCI OpenSearch	READ
`create_hcm_employee`	🔵 Private	HR-Agent only	OIC → Fusion HCM REST	WRITE-IRREVERSIBLE
`create_payroll_element`	🔵 Private	Finance-Agent only	OIC → Fusion Payroll REST	WRITE-IRREVERSIBLE
`provision_saas_apps`	🔵 Private	IT-Agent only	OIC → ITSM REST API	WRITE-REVERSIBLE
`assign_desk_and_kit`	🔵 Private	Facilities-Agent only	OIC → Facilities REST	WRITE-REVERSIBLE

🔗 Section 8: How everything works together — the full picture

Now let's see all the components of a Multi-Agent Architecture working together in a single end-to-end view. This is the mental model every OIC architect must be able to draw on a whiteboard.

🏛️ OIC Multi-Agent Architecture — complete reference

── TRIGGER ──

👤 HR Manager clicks "Onboard New Employee" in Fusion Self-Service portal
OIC REST Trigger receives: employee details + start date + role + BU

⬇️

── OIC PROJECT BOUNDARY ──

👑 ORCHESTRATOR AGENT — OnboardOrchestrator

🧠 OCI GenAI LLM

📝 Orchestrator Prompt Template

🔄 ReAct Loop Controller

📋 Session State

Analyses goal → Creates delegation plan → Calls Sub-Agents → Assembles results

⬇️

Parallel Batch 1

⬇️

Parallel Batch 1

⟳

waits for IDs

⟳

waits for IDs

👤

HR-Agent

Fusion HCM
Creates profile
Returns employee_id

💻

IT-Agent

ITSM + AD
Provisions access
Returns ticket_id

💰

Finance-Agent

Fusion Payroll
Sets up payroll
Returns payroll_id

🏢

Facilities-Agent

FM System
Assigns desk
Returns desk_id

⬇️ All results collected by Orchestrator

✅ ONBOARDING COMPLETE: Orchestrator assembles all results → sends Teams summary to HR Manager → writes to OCI Audit → session ends with SUCCESS status.

📚 OCI OpenSearch
Shared Knowledge Base

💾 OCI Object Storage
Prompt Templates + Memory

📊 OCI Logging
Observability + Audit

🛡️ OCI IAM + Vault
Security Foundation

🏭 Section 9: Real corporate example — InfraGroup India

Company: InfraGroup India — infrastructure conglomerate with 12,000 employees, ₹15,000 Cr turnover, running Oracle Fusion HCM + Finance + SCM on OIC.

Challenge: 300–500 new hires monthly. Manual onboarding took 5 business days average. 23% error rate in payroll setup due to manual data re-entry. 40% of IT access tickets were delayed. New employees frequently started without system access on Day 1.

Solution: Multi-Agent Onboarding System built as a single OIC Project — "ONBOARD-PRJ-v1.0" — with 1 Orchestrator and 4 Sub-Agents.

🏭 ONBOARD-PRJ-v1.0 — complete architecture

Agent	Type	OCI GenAI model	max_iterations	Tools count	Avg completion time
OnboardOrchestrator	👑 Orchestrator	Cohere Command R+	50	7 (4 sub-agents + 3 shared)	4 min 18 sec total
HR-Agent	🤝 Sub-Agent	Cohere Command R	15	4 private + 3 shared	1 min 12 sec
Finance-Agent	🤝 Sub-Agent	Cohere Command R	15	4 private + 3 shared	1 min 45 sec
IT-Agent	🤝 Sub-Agent	Cohere Command R	20	4 private + 3 shared	2 min 05 sec
Facilities-Agent	🤝 Sub-Agent	Cohere Command R	12	4 private + 3 shared	0 min 58 sec

5 days

→ 4 min 18 sec average

23%

Payroll error rate → 0% (no manual re-entry)

100%

Day-1 system access rate (was 60%)

3 FTE

HR team time freed for strategic work

🛠️ Section 10: Step-by-step implementation in OIC Gen3

Here is the exact sequence to build a Multi-Agent system in OIC. Follow this order — skipping phases causes integration issues that are very hard to debug later.

Create the OIC Gen3 Project

In OIC Gen3 console → Projects → Create Project. Name it clearly (e.g., ONBOARD-PRJ). Set environment: DEV. Add team members with DEVELOPER role. This project becomes the container for everything. Create all OIC connections at the project level — Fusion HCM Adapter, Fusion Payroll Adapter, ITSM REST Adapter, Teams Webhook, OCI OpenSearch. All agents inside this project inherit access to these connections.

Build and test all OIC tool integrations

Inside the Project, create each OIC REST-triggered integration that will back a tool. For shared tools first: create_audit_log, send_teams_notification, search_policy_kb. Then private tools per agent domain. Every integration must: retrieve secrets from OCI Vault (not hardcoded), return SUMMARY payloads (no full Fusion JSON), handle errors gracefully and return a structured error JSON. Test each integration individually with real sandbox data before proceeding. 100% pass rate mandatory.

Write all prompt templates (one per agent)

Write 5 prompt templates: 1 for Orchestrator + 1 each for HR, Finance, IT, Facilities Sub-Agents. Each follows the 7-section structure (Identity, Mandate, Hard Rules, Tool Guidance, Reasoning Protocol, Completion Signal, Output Format). The Orchestrator template focuses on coordination logic; Sub-Agent templates focus on domain depth. Upload all 5 to OCI Object Storage. Version them: orchestrator_prompt_v1.0.txt, hr_agent_prompt_v1.0.txt, etc.

Build and register each Sub-Agent (bottom up)

In AI Agent Studio within the Project → New Agent. Build Sub-Agents first, before the Orchestrator. For each Sub-Agent: select LLM (Cohere Command R for Sub-Agents — faster and cheaper than Command R+), set max_iterations=15, temperature=0.1, session_timeout=180s. Point to its prompt template URL. Register its private tools + shared tools. Assign a narrow OCI IAM role. Test each Sub-Agent independently in the Studio test panel using realistic sample inputs from the Orchestrator.

Build the Orchestrator Agent last

In AI Agent Studio → New Agent → type: Orchestrator. Select OCI GenAI (Cohere Command R+ — the Orchestrator needs the strongest reasoning). max_iterations=50, temperature=0.1, session_timeout=600s. Point to orchestrator prompt template. Register tools: each Sub-Agent appears as a tool (its REST invoke endpoint), plus the 3 shared tools. Assign an OCI IAM role that can invoke all Sub-Agent endpoints but no direct Fusion APIs — the Orchestrator only calls Sub-Agents, never systems directly.

Configure guardrails at every level

In OIC BPM: create Human Approval gates on all WRITE-IRREVERSIBLE tools (create_hcm_employee, create_payroll_element) — even inside Sub-Agents, the BPM gate exists in the backing OIC integration. In OCI API Gateway: set rate limits per agent (Sub-Agents: 30 tool calls/session max; Orchestrator: 60 tool calls/session max). In OCI CloudGuard: alert on any agent making 50+ tool calls in one session. PII masking: salary_band, bank details must be masked in Finance-Agent and HR-Agent tool responses before they enter LLM context.

Create trigger and set up observability

In the Project: create an OIC REST-triggered integration that is the entry point — receives the HR Manager's form submission, validates input, builds the goal JSON, and invokes the Orchestrator Agent. Also create an OIC Scheduled Integration for batch processing (Monday morning bulk onboarding). Configure OCI Logging to capture all 5 agent session traces under the same Project correlation ID, so one search shows the entire onboarding workflow trace. Build an OCI Monitoring dashboard with: total onboarding time per request, success rate per agent, token cost per onboarding, and error rate by agent.

Deploy the Project as one unit

In OIC Projects → ONBOARD-PRJ → Deploy to TEST environment. The entire project — all 5 agents, all tools, all prompt templates, all connections, all schedules — deploys together as one versioned unit. Test with 5 synthetic employee records in TEST. Pass all scenarios. Then deploy to PRODUCTION using the same Project deploy action. Never deploy individual agents independently — always deploy the whole Project to maintain consistency between Orchestrator and Sub-Agents.

🧪 Section 11: Testing and validation

🔧 Phase 1 — Tool unit tests

Test every OIC integration (tool backend) independently. Verify correct Fusion connectivity, SUMMARY payload format, error handling, timeout behaviour, and OCI Vault secret retrieval. 100% pass rate before any agent testing begins.

🤝 Phase 2 — Sub-Agent isolation tests

Test each Sub-Agent completely independently — as if the Orchestrator is calling it. Send structured JSON inputs matching exactly what the Orchestrator will send. Verify structured JSON outputs match exactly what the Orchestrator expects. Test error scenarios: Fusion API down, invalid input, missing fields.

👑 Phase 3 — Orchestrator tests

Test the Orchestrator with mock Sub-Agent responses (stub the Sub-Agent endpoints). Verify correct delegation sequence, correct parallel vs sequential logic, correct handling of Sub-Agent errors, and correct final output assembly. Then run with real Sub-Agents connected end-to-end.

🎭 Phase 4 — Full end-to-end scenarios

Run 6 full scenarios: (1) Happy path — all agents succeed, (2) HR-Agent failure — Orchestrator handles gracefully, (3) Finance-Agent timeout — retry logic fires, (4) Parallel batch race condition — later batch runs first, (5) All agents fail — Orchestrator escalates, (6) Load test — 10 concurrent onboardings.

🛡️ Phase 5 — Security tests

Test: prompt injection via malicious employee name field. Test: HR-Agent attempting to call Finance tools (must fail). Test: Finance-Agent accessing another BU's salary data (must fail). Test: Orchestrator skipping human approval gate on irreversible tools (must be blocked at OIC BPM layer).

🚨 Section 12: Common mistakes in Multi-Agent Architecture

💥 Mistake 1 — building the Orchestrator before the Sub-Agents

Without tested Sub-Agents, you cannot define the Orchestrator's tool catalogue correctly. You'll define the wrong input/output contracts and have to redo the Orchestrator. Fix: always build Sub-Agents first, test them fully, then build the Orchestrator that calls them. Bottom-up, not top-down.

💥 Mistake 2 — Sub-Agents returning unstructured text to the Orchestrator

If the HR-Agent returns "Employee John Smith was created with ID EMP-00823" as free text, the Orchestrator must parse this string to extract the employee_id. One format variation and the whole workflow breaks. Fix: every Sub-Agent output format must be a strict JSON schema: {"status":"SUCCESS","employee_id":"EMP-00823","profile_url":"..."}

💥 Mistake 3 — using a large expensive model for Sub-Agents

Sub-Agents do focused, simple tasks. Using Cohere Command R+ (the most powerful model) for every Sub-Agent multiplies your LLM cost by 5. Fix: use Cohere Command R (faster, cheaper) for Sub-Agents. Reserve Command R+ for the Orchestrator which needs the strongest coordination reasoning. Right-size the model to the task complexity.

💥 Mistake 4 — no correlation ID across agent sessions

When troubleshooting a failed onboarding, you see 5 separate agent session logs in OCI Logging with no connection between them. You cannot tell which HR-Agent session belongs to which Orchestrator run. Fix: the Orchestrator generates a workflow_correlation_id at session start and passes it as a parameter to every Sub-Agent invocation. Sub-Agents include this ID in every log entry and their final response. One search for the correlation ID shows the entire workflow trace across all 5 agents.

💥 Mistake 5 — deploying agents individually instead of as a Project

Deploying the Orchestrator v2 while Sub-Agents are still on v1 breaks the input/output contracts. Partial deployment causes subtle, hard-to-debug failures in production. Fix: always deploy the entire OIC Project as one atomic deployment. If even one component fails deployment validation, the whole project stays on the previous version. Never deploy components individually.

💥 Mistake 6 — Sub-Agents that are too big (trying to do too much)

An HR-Agent that also validates employment eligibility, runs background checks, and sends welcome emails is no longer a specialist — it is a small Orchestrator. It will hit context limits and fail on complex cases. Fix: if a Sub-Agent needs to do more than 3–4 steps, split it. Create a second Sub-Agent. Let the Orchestrator coordinate two HR-related agents instead of one overloaded one.

🏆 Section 13: Best practices for Multi-Agent systems

👑 Orchestrator best practices

Orchestrator should NEVER call Fusion APIs directly — delegate to Sub-Agents
Orchestrator prompt template focuses on coordination, not domain knowledge
Use Command R+ for Orchestrator (complex coordination reasoning needed)
Always define explicit parallel vs sequential rules in the Orchestrator prompt
Generate and pass workflow_correlation_id to every Sub-Agent call
Handle Sub-Agent errors gracefully — partial success is better than total failure

🤝 Sub-Agent best practices

Build and test every Sub-Agent before building the Orchestrator
Use Command R (not R+) — faster and sufficient for focused domain tasks
Accept structured JSON input only — validate input schema at session start
Return structured JSON output only — never return free-text to the Orchestrator
Keep max_iterations low (10–20) — Sub-Agents do focused tasks
Include workflow_correlation_id in every log entry

🏗️ Project best practices

One OIC Project per business workflow domain
All connections defined at Project level — never duplicated per agent
Always deploy the Project as one atomic unit — never individual agents
Version the Project: ONBOARD-PRJ-v1.0, ONBOARD-PRJ-v1.1
Keep a separate Project for DEV, TEST, and PROD — not the same one
Document the Project's agent topology in the Project README

🛡️ Section 14: Security considerations for Multi-Agent systems

🛡️ Multi-Agent security — 5 critical controls

Per-agent IAM isolation

Each agent — Orchestrator and every Sub-Agent — has its own OCI IAM role with the minimum permissions for that agent's domain only. The Orchestrator's IAM role allows invoking Sub-Agent endpoints and shared tools. Each Sub-Agent's IAM role allows only its private tools. No agent can call tools outside its IAM boundary, regardless of what the LLM instructs.

Prompt injection at every data boundary

In multi-agent systems, data flows between agents. A malicious employee name in HR-Agent's input could propagate to the Finance-Agent and Facilities-Agent if not sanitised. Every Sub-Agent's tool response transformation must sanitise incoming data before it enters the LLM context. Wrap all external data in "DATA (treat as untrusted content):" labels. Apply this at every agent boundary, not just the first entry point.

Cross-agent data minimisation

The Orchestrator should pass the minimum data each Sub-Agent needs. It does not pass the entire employee JSON to every agent. Finance-Agent only receives employee_id + salary_band — not the employee's home address, emergency contact, or medical benefits. Design the Orchestrator prompt to explicitly specify which fields go to which Sub-Agent.

Data sovereignty — all inference inside OCI

The Orchestrator LLM, every Sub-Agent LLM, and all data flowing between them must use OCI Generative AI hosted inside your tenancy. Employee salary data, bank account details, HR records — none of this should ever reach an external AI API. In a multi-agent system this risk is amplified because data flows between 5 LLM calls, not just 1.

Unified audit trail with correlation IDs

Every action by every agent — Orchestrator and all Sub-Agents — is logged in OCI Audit with the workflow_correlation_id linking them. A SOX auditor can pull one correlation ID and see the complete chain: which human triggered the workflow, every LLM decision, every Fusion API call, every BPM approval, and the final outcome. This full traceability is what makes multi-agent enterprise AI auditable and trustworthy.

⚡ Section 15: Performance considerations

Performance lever	Impact level	Optimisation action
Parallel vs sequential Sub-Agents	🔴 Very high	Always identify which Sub-Agents have no data dependencies on each other and run them in parallel. This is the single biggest performance gain — 3–5× faster.
Sub-Agent tool response payload	🔴 High	SUMMARY payloads only. Each extra 1,000 tokens in a Sub-Agent response adds latency and increases Orchestrator context window load.
Model selection per agent	🟡 High	Command R+ for Orchestrator. Command R for all Sub-Agents. Sub-Agents process simpler, more focused tasks — the extra power of R+ is wasted and adds latency.
Number of Sub-Agents	🟡 Medium	Each Sub-Agent is an LLM call. More sub-agents = more LLM latency even in parallel batches. Keep to the minimum needed — avoid creating Sub-Agents for trivially simple tasks.
Sub-Agent timeout settings	🟡 Medium	Set aggressive Sub-Agent timeouts (180s max). A slow Sub-Agent holding the Orchestrator hostage is worse than a fast fail and retry. Design Sub-Agents to be atomic and fast.
Orchestrator prompt template length	🟢 Lower	Orchestrator template loads once and persists. Keep under 1,200 tokens. Precise coordination rules are better than verbose ones — the Orchestrator's job is clear delegation, not deep analysis.

🚀 Section 16: Production considerations

🔄 Partial success handling

When 3 of 4 Sub-Agents succeed and 1 fails, the Orchestrator must not discard all the work done. Define a PARTIAL_SUCCESS status. Log what succeeded, flag what failed, notify the human manager with the partial result and the specific failure. A human can then manually complete the one failed step rather than restarting the entire onboarding.

📦 Idempotency design

If the Orchestrator is retried after a partial failure, it must not create duplicate HCM employees or payroll elements. Every Sub-Agent tool that creates data must be idempotent — if the employee already exists with that national ID, return the existing record instead of creating a duplicate. Design idempotency keys at the tool level.

📊 Per-agent cost tracking

Track OCI GenAI token cost per agent per workflow run. You will discover that one Sub-Agent consumes 60% of the project's tokens. Optimise that agent first. Set monthly budget alerts per project. Total LLM cost per onboarding should be a KPI on your team dashboard.

📈 Progressive rollout

In the first 2 weeks of production: run the multi-agent system in shadow mode — it processes onboardings but humans also do them manually. Compare results. When the agent matches or exceeds human accuracy on 20 consecutive onboardings, switch to fully automated mode. Never go from zero to 100% automation on day one.

🎯 Section 17: Key takeaways

🏗️ Architecture takeaways

Multi-Agent is for complex workflows that exceed a single agent's context, speed, or domain scope
OIC Gen3 Project is the container — deploy everything as one atomic unit
One Orchestrator coordinates many focused Sub-Agents
Orchestrator delegates — it never does domain work itself
Parallel Sub-Agents = 3–5× faster workflows — always exploit this

🔧 Implementation takeaways

Build Sub-Agents first, Orchestrator last
Sub-Agents must accept and return strict JSON schemas
Use Command R+ for Orchestrator, Command R for Sub-Agents
Workflow correlation ID links all agent sessions for tracing
Tool integrations must be idempotent — retries cannot create duplicates

🛡️ Security takeaways

Every agent has its own narrow IAM role — no agent calls tools outside its boundary
Sanitise data at every agent boundary — injection risks multiply across agents
Pass minimum data between agents — data minimisation is multi-agent security
All LLM inference on OCI GenAI inside tenancy — no external AI APIs
Unified audit trail with correlation IDs across all 5 agent sessions

🎓 Section 18: Interview questions — with senior-level model answers

Q1: What is an OIC Project and why is it essential for Multi-Agent Architecture?

Model answer: An OIC Project is a unified workspace that groups all related integration artifacts — agents, tools, prompt templates, connections, and schedules — into one manageable, deployable, version-controlled unit. For multi-agent architecture specifically, Projects solve three critical problems: they ensure atomic deployment (all agents deploy together, preventing version mismatches), they enable shared connections (one Fusion adapter connection shared across all agents with no duplication), and they provide unified version control (rolling back the project rolls back the entire agent team, not just individual members). Without Projects, multi-agent systems become impossible to maintain reliably in production environments.

Q2: What is the difference between an Orchestrator Agent and a Sub-Agent in OIC?

Model answer: The Orchestrator Agent is the coordinator — it receives the high-level business goal, analyses dependencies between sub-tasks, delegates to domain specialists, manages parallel vs sequential execution, and assembles final results. It never performs domain-specific work itself. Sub-Agents are domain specialists — each owns one business domain, has deep expertise in that domain's tools, and executes a specific scoped task to completion before returning a structured result to the Orchestrator. Key differences: the Orchestrator uses a stronger LLM (Command R+) with higher max_iterations and a coordination-focused prompt; Sub-Agents use a lighter LLM (Command R), lower max_iterations, and a domain-expert prompt. Sub-Agents appear as tools in the Orchestrator's tool catalogue.

Q3: How do you handle a scenario where one Sub-Agent fails while others succeed in a Multi-Agent workflow?

Model answer: Production multi-agent systems must define a PARTIAL_SUCCESS state explicitly in the Orchestrator's prompt template and logic. When a Sub-Agent returns an error, the Orchestrator should: (1) log the failure with the workflow_correlation_id, (2) continue collecting results from other Sub-Agents that were running in parallel, (3) assemble a partial result summarising what succeeded, (4) flag the failed component with the specific error, and (5) notify the designated human manager with both the partial completion status and the actionable failure detail. The key principle is that the work already successfully completed by other Sub-Agents is not discarded — the human manager only needs to manually handle the one failed component, not restart the entire workflow. Idempotency in tool design is critical here — if the Orchestrator retries the failed Sub-Agent, it must not create duplicates in systems where data was already partially created.

Q4: Why should the Orchestrator Agent use Cohere Command R+ while Sub-Agents use Cohere Command R?

Model answer: Model selection should match task complexity to cost and latency requirements. The Orchestrator performs complex coordination reasoning — it must understand data dependencies between agents, determine which tasks can run in parallel, handle partial failures, assemble results from multiple sources, and make nuanced routing decisions. This requires the strongest model available. Sub-Agents, by contrast, perform focused, single-domain tasks with clear inputs and outputs. An HR-Agent creating a Fusion HCM employee record is a well-defined, bounded task that does not require frontier-level reasoning. Using Command R+ for Sub-Agents multiplies LLM cost by 4–5× with no quality improvement, because the task complexity doesn't justify it. The right principle is: use the minimum model capable of doing the job reliably, not the most powerful model available.

Q5: How do you ensure full traceability and auditability across a 5-agent workflow in production?

Model answer: The key mechanism is a workflow_correlation_id generated by the Orchestrator at session start. This ID is passed as a parameter to every Sub-Agent invocation and must be included in every log entry by every agent throughout the workflow. In OCI Logging, a single search for the correlation ID returns the complete workflow trace — Orchestrator decisions, every Sub-Agent session, every tool call, every Fusion API interaction, every BPM approval, and the final outcome — all linked. This creates a three-way audit chain: Orchestrator session ID → Sub-Agent session IDs → OIC tracking instances → Fusion transaction IDs. For SOX compliance specifically, every Fusion transaction created by any agent in the workflow must carry the identity of the original human user who triggered the Orchestrator, not the agent service account — this is enforced through OCI IAM identity propagation configured at session start.

Q6: A client says they want to build one "super agent" that handles their entire employee lifecycle — hiring, payroll, training, performance, and offboarding — as a single agent. What would you tell them?

Model answer: I would strongly recommend against this approach for three specific, architectural reasons. First, context window explosion: a single agent handling all five lifecycle stages needs every domain's tools, policies, and data loaded simultaneously. The context fills up quickly, older information gets dropped, and reasoning quality degrades — especially on edge cases that span multiple domains. Second, domain confusion: when the same agent handles both "approve a hire" and "approve a performance rating" and "approve an offboarding," the boundaries between these approval types blur in the LLM reasoning. One domain's rules contaminate another's decision-making. Third, no parallelism: a super agent works sequentially by definition — payroll cannot be processed while training is being set up. A properly architected Multi-Agent system with an Onboarding Orchestrator, Training Orchestrator, and Performance Orchestrator — each with their own specialised Sub-Agents — runs in parallel where dependencies allow, scales independently, and can be evolved without touching other lifecycle stages. I would propose a phased approach: build the Onboarding Multi-Agent system first, validate it in production, then extend the architecture to other lifecycle stages using the same Project pattern.

Q7: In an employee onboarding multi-agent system, the HR-Agent needs to pass the employee_id to the Finance-Agent and Facilities-Agent. What is the correct way to design this data flow and why?

Model answer: The correct design is for the Orchestrator to be the single data hub — it receives the employee_id from HR-Agent in a structured JSON response, stores it in its Session State, and then explicitly passes it as a parameter in the structured JSON input when invoking Finance-Agent and Facilities-Agent. Sub-Agents should never call other Sub-Agents directly — all data flow goes through the Orchestrator. This design is correct for three reasons: (1) it preserves the Orchestrator's coordination role and full visibility over the workflow state, (2) it enforces data minimisation — Finance-Agent receives only employee_id + salary_band, not the complete HR profile, and (3) it maintains clear error boundaries — if Finance-Agent fails because it received a null employee_id, the root cause is immediately traceable to the HR-Agent result in the Orchestrator's session log. The anti-pattern would be HR-Agent directly calling Finance-Agent with the employee_id — this creates hidden dependencies between Sub-Agents that the Orchestrator cannot observe, control, or trace.

🌱 Final word from the architect:

Multi-Agent Architecture is the pattern that takes OIC Gen3 from "smart automation" to "enterprise-grade autonomous intelligence." Once you understand how the Orchestrator coordinates its team of specialists inside an OIC Gen3 Project, you will see opportunities to apply this pattern everywhere — procurement, finance close, customer onboarding, incident response, contract management.

A Multi-Agent Architecture in OIC Gen3 is not just a more powerful AI system. It is a fundamentally better way to design complex enterprise workflows — one that reflects how the best human teams actually work. Specialists doing what they're best at. A coordinator keeping everyone aligned. Clear handoffs. Shared resources. Parallel execution. A single traceable audit trail.

The key insight that separates architects from beginners is this: you are not building one AI agent. You are designing a team of AI specialists, defining how they collaborate, what they can do, what they must never do, and how the organisation can trust and audit every decision they make. That is engineering. That is architecture.

Start with one workflow. Build it right. Earn trust through observability. Then scale.

Coordinate with Intelligence. Specialise with Precision. Deliver with Trust. 👑 🤝

Search This Blog