Skip to main content

Design Multi-Agent Systems: Deploy and Operate Multi-Agent AI Systems – Production Release, Rollback and Recovery

Calculating read time…

Imagine an Accounts Payable team has successfully tested its invoice investigation and policy interpretation agents. Both specialists produce the expected recommendations in development. The orchestration integration also completes successfully. The team now promotes the solution to another OIC instance.

However, the target environment has different connection credentials, the imported project cannot activate its AI agent, and two cases are still waiting for asynchronous tool responses from the previous release. The project import succeeded, but that does not mean the business application is ready. Production readiness requires more than moving artifacts: it requires compatible versions, environment-specific configuration, controlled activation, recoverable case state, and observable operational ownership. This tutorial rehearses that complete release and recovery process for an OIC-based multi-agent solution.

01. The Release You Will Prepare and Rehearse

The practical outcome

You will prepare a versioned release of an Invoice Exception Resolution solution, promote its OIC project artifacts into a separate test instance, re-establish the target environment's dependencies, validate AI agent activation, and rehearse containment and recovery after introducing a deliberately faulty release.

What kind of tutorial is this?

This is a release-engineering rehearsal playbook, not a click-by-click lab that builds the entire OIC solution from an empty instance. It assumes that the development project, specialist agents, thinking pattern, and tool integrations already exist. It supplies synthetic case fixtures, illustrative request/response contracts, promotion gates, and recovery records. Tenant-specific endpoint URLs, credentials, agent configuration values, and deployment evidence must come from your own OIC environments.

The finished engineering package will contain three principal deliverables:

  • A release manifest describing the complete solution and its environment dependencies.
  • A launch checklist recording promotion order, verification evidence, and approval gates.
  • A short incident runbook covering intake suspension, manual processing, rollback, and in-flight case protection.

The baseline remains read-only. The specialists investigate invoice evidence and interpret policy; OIC coordinates their work and produces a recommendation. An authorized transaction integration is outside this release.

Understand the multi-agent architecture before promoting it

A multi-agent solution is not simply one large agent with several prompts. In this example, separate components have separate responsibilities:

  1. OIC orchestration integration: Receives the business case, calls the specialist agents, validates their outputs, manages the workflow, and records the final case recommendation. It is an OIC integration; it does not automatically become a third AI agent.
  2. Invoice investigation agent: Focuses on invoice, purchase-order, receipt, quantity, duplicate, and price evidence. It uses its registered tool integrations to retrieve that evidence.
  3. Policy interpretation agent: Focuses on the applicable Accounts Payable policy and interprets the already validated finding against that policy.
  4. Agentic AI tool integrations: OIC integrations exposed to an AI agent as callable tools. The integrations perform the bounded enterprise operation, such as retrieving evidence or policy data. They are not additional AI agents.
  5. Callback integration, when an asynchronous tool is used: Receives completion information and uses the OIC AI Agent action configured for Callback AI agent to return the tool result to the waiting conversation. This callback receiver is distinct from the asynchronous tool integration itself.

A thinking pattern is a configured reasoning approach that an agent uses; it is not another specialist or an orchestration integration. Each specialist has its own role and tool permissions, while the OIC coordinator remains responsible for validating the combined business result.

Conceptual request and response flow

Business case → OIC orchestration → invoice investigation agent → evidence tool integration

OIC orchestration → policy interpretation agent → policy tool integration

OIC validates both specialist results → records the case recommendation → routes exceptions to the authorized human process

See what a production smoke test must prove

Use the following synthetic test case throughout the release rehearsal.

Test case: CASE-1001

Invoice: INV-1042

Purchase order: PO-2087

Invoice line: ITEM-A, quantity 10, unit price USD 125.00.

Purchase order line: ITEM-A, quantity 10, unit price USD 120.00.

Receipt: RCV-7721, quantity received 10.

Previous invoicing: No duplicate invoice and no previously invoiced quantity.

Illustrative policy: Price variance up to 2% is tolerated. A higher variance requires Accounts Payable review.

For this rehearsal, send the following illustrative JSON request to the REST trigger exposed by CASE_ORCHESTRATOR. The target URL and authentication are environment-specific; use the endpoint configured in your TEST instance rather than copying a development URL.

{
  "caseId": "CASE-1001",
  "invoiceId": "INV-1042",
  "purchaseOrderId": "PO-2087",
  "receiptId": "RCV-7721"
}

Load the synthetic invoice, purchase-order, receipt, and policy records into the TEST data source, or configure read-only mock tool integrations to return the fixtures shown below. The orchestration integration passes the identifiers to the evidence lookup. The deterministic evidence integration calculates the price variance using the purchase-order unit price as the denominator:

(USD 125.00 − USD 120.00) ÷ USD 120.00 × 100 = 4.1667%, rounded to 4.17%.

{
  "caseId": "CASE-1001",
  "invoiceId": "INV-1042",
  "purchaseOrderId": "PO-2087",
  "receiptId": "RCV-7721",
  "quantityReceived": 10,
  "invoiceQuantity": 10,
  "quantityCheck": "MATCH",
  "invoiceUnitPrice": 125.00,
  "purchaseOrderUnitPrice": 120.00,
  "priceVariancePct": 4.17,
  "duplicateCheck": "CLEAR",
  "deterministicFinding": "PRICE_VARIANCE"
}

The investigation specialist returns the supporting evidence and its source references:

{
  "caseId": "CASE-1001",
  "specialist": "InvoiceInvestigationAgent",
  "finding": "Quantity matches, but invoice unit price is higher",
  "evidence": [
    "INV-1042: invoice line 10",
    "PO-2087: purchase order line",
    "RCV-7721: goods receipt"
  ],
  "checks": {
    "quantity": "MATCH",
    "duplicate": "CLEAR",
    "priceVariancePct": 4.17
  },
  "recommendation": "PRICE_VARIANCE_REQUIRES_REVIEW"
}

The coordinator passes the policy specialist a compact, validated input. This is an application-defined example contract, not an Oracle-reserved payload:

{
  "caseId": "CASE-1001",
  "investigation": {
    "finding": "PRICE_VARIANCE",
    "quantityCheck": "MATCH",
    "duplicateCheck": "CLEAR",
    "observedVariancePct": 4.17,
    "evidenceReferences": ["INV-1042", "PO-2087", "RCV-7721"]
  },
  "policy": {
    "policyReference": "AP-PRICE-02",
    "tolerancePct": 2.00,
    "rule": "Price variance above 2% requires Accounts Payable review"
  }
}

The policy specialist evaluates that input and returns the following response:

{
  "caseId": "CASE-1001",
  "specialist": "PolicyInterpretationAgent",
  "policyReference": "AP-PRICE-02",
  "tolerancePct": 2.00,
  "observedVariancePct": 4.17,
  "decision": "HOLD_FOR_AP_REVIEW",
  "reason": "Price variance exceeds the permitted tolerance"
}

The orchestration integration validates that both responses belong to CASE-1001, that the required evidence exists, and that the recommendation agrees with the deterministic matching result.

Expected final business response

Case: CASE-1001

Invoice: INV-1042

Purchase order: PO-2087

Investigation: Quantity and receipt match; price variance is 4.17%.

Policy: AP-PRICE-02 allows a 2% variance.

Final recommendation: Hold for Accounts Payable review.

Transaction executed: No. No invoice payment, posting, or approval has been performed.

This is an illustrative release smoke-test interaction, not a report of an executed tenant run. Its purpose is to define the observable business result that the promoted release must reproduce. CASE-1001 is the positive business-result fixture. The two asynchronous cases mentioned in the opening scenario are separate recovery fixtures, CASE-1002 and CASE-1003; they must not be represented as pending callbacks on CASE-1001.

02. In This Post

  1. The release and expected outcome
  2. Before you begin
  3. Production release concepts and decisions
  4. Practical release and recovery rehearsal
  5. Test and prove the release
  6. Troubleshooting and recovery
  7. Enterprise operating controls
  8. Completion checklist
  9. Five useful FAQs
  10. Verified references
  11. Finally — Summary

03. Before You Begin

The following are the dependencies for the release rehearsal, rather than introductory OIC prerequisites.

  • Development OIC instance: Contains the source project, integrations, thinking patterns, specialist agents, and associated configurations.
  • Separate test OIC instance: Receives the promoted release and has independent connections, credentials, permissions, and synthetic test data.
  • Release access: An authorized developer or administrator who can export and import projects, configure connections, and activate integrations and AI agents.
  • Source control: A repository or controlled release location for the CAR archive, source definitions, prompt versions, manifest, and checksums.
  • Case persistence: An external durable case store or an established application-level state service if the solution supports long-running cases.
  • Test evidence: Invoice, purchase-order, receipt, and policy fixtures, plus representative expected results.
  • Agentic AI tool integrations: Each tool integration is in an OIC project, is Active, and meets Oracle's documented tool requirements: a REST Adapter trigger, POST operation, and JSON request and response payloads.
  • Asynchronous callback path, if used: An asynchronous tool integration, a durable store for callback correlation data, and a separate callback integration configured with the AI Agent action.

The identifiers, manifest values, case records, and JSON bodies in this article are synthetic examples. They define the contracts and evidence that a release rehearsal should use; they do not represent a previously completed tenant deployment or executed runtime test.

04. Production Release Concepts That Affect This Solution

4.1 A release is larger than an OIC project archive

A release is the complete, identifiable collection of resources and configuration needed to reproduce a business capability in a target environment.

For a multi-agent solution, that includes:

  • OIC project, integration identifiers, and exact integration versions.
  • Agent definitions, thinking patterns, instructions, prompt templates, and attached tools.
  • Agent tool integrations and their request/response contracts.
  • Connections, connection properties, certificates, authentication identities, and LLM endpoints.
  • External case store, case schema, state transitions, and persisted business data.
  • Approval definitions and authorized transaction boundaries, when present.
  • Monitoring configuration, correlation mappings, operational alerts, and support documentation.

The CAR file is a promotion artifact, not a complete backup of all these dependencies. Oracle explicitly documents that remote shared connections and local connection override configurations or associations are not included in a project export. Imported connections also require environment-specific configuration.

4.2 Deployment compatibility is a relationship between versions

Versioning only the orchestration integration is insufficient. A change in the investigation agent's output, the policy contract, or a prompt template can break a coordinator even when the OIC integration itself activates successfully.

Give the complete solution its own release identifier, for example invoice-exception-resolution 1.4.0. Separately record the versions of individual OIC integrations, agents, patterns, prompts, and external contracts.

Persisted case state must also carry a schema version. An in-flight case created under schema version 1 should not silently be interpreted using version 2 fields if their meaning has changed.

4.3 Environment promotion must not promote production secrets

Development, test, and production should have independent connection credentials, LLM credentials, target endpoints, data, and access permissions. Use environment-specific secret references and identities rather than embedding credentials in prompts, source control, or exported configuration.

The same logical connection name can exist in multiple instances, but its endpoint, authentication identity, and permissions must be appropriate to that environment. A read-only investigation identity must not become a privileged production posting identity simply because the project was imported.

4.4 In-flight work is a separate release dependency

An in-flight case is a business case that has started but has not reached a terminal status. It might be waiting for a specialist, a tool callback, a policy decision, or an authorized human approval.

Persist a release reference and case schema version with every such case. Also preserve the agent conversation identifiers, callback correlation information, and last completed orchestration step where those features are used.

A conversation ID identifies an AI-agent conversation. It is not, by itself, a durable business case record, an authenticated user identity, or proof of authorization.

4.5 Rollback does not reverse a business transaction

Deployment rollback restores application artifacts and routing to a compatible release. Business compensation reverses or corrects an already completed external business operation through an explicitly authorized process.

For example, restoring an earlier orchestration integration does not undo an invoice posting that has already committed in an ERP system. Those are separate operations with separate evidence and authorization requirements.

A simple analogy

Think of a production release as handing an Accounts Payable case file to a different team. The receiving team needs the correct documents, policy version, access rights, and record of unfinished work. Copying the folder does not recreate those responsibilities or authorize a payment.

05. Practical Work — Release, Promote, Validate and Recover

Phase 1 — Create the complete release inventory

Starting point: The development OIC project and its source-controlled definitions.

Do not begin by exporting the project and assuming that the archive identifies everything needed in production. First inventory the full dependency chain.

Release inventory and version records
Inventory area Record
OIC integrationsProject, integration code, version, dependencies and activation state.
AI agentsAgent identifier, version, role, guidelines, pattern dependency and attached tools.
Thinking patternsPattern identifier, recipe, version, LLM configuration and activation requirement.
Prompts and templatesExact content, version or source-control reference and checksum.
Tools and contractsTool integration, request schema, response schema, version and identity requirements.
ConnectionsConnection name, endpoint ownership, credential reference, certificates and test status.
Case stateState-store location, schema version, retention, backup, in-flight case count.
Approvals and transactionsApproval workflows, authorization boundaries and transaction integrations, if applicable.
OperationsMonitoring, alert owners, runbooks, support contacts and escalation rules.

Create a release manifest and commit it alongside the CAR file. The following is an illustrative starting structure; replace sample identifiers with the actual artifacts.

release:
  name: invoice-exception-resolution
  version: "1.4.0"
  sourceCommit: "replace-with-git-commit"
  artifact:
    file: "AP_MULTI_AGENT-1.4.0.CAR"
    sha256: "record-actual-checksum"

  oic:
    product: "Oracle Integration 3"
    documentationChecked: "2026-11-18"
    sourceInstance: "OIC-DEV"
    targetInstance: "OIC-TEST"

  project:
    code: "AP_MULTI_AGENT"
    deployment: "AP_READ_ONLY_1_4_0"

  integrations:
    - code: "CASE_ORCHESTRATOR"
      version: "01.04.0000"
    - code: "INVOICE_EVIDENCE_LOOKUP"
      version: "01.02.0000"
    - code: "POLICY_LOOKUP"
      version: "01.02.0000"

  agents:
    - code: "INVOICE_INVESTIGATION"
      version: "01.04.0000"
      pattern: "AP_REACT_PATTERN"
      roleGuidelinesDigest: "record-checksum"
      tools:
        - "INVOICE_EVIDENCE_LOOKUP"
    - code: "POLICY_INTERPRETATION"
      version: "01.02.0000"
      pattern: "AP_REACT_PATTERN"
      roleGuidelinesDigest: "record-checksum"
      tools:
        - "POLICY_LOOKUP"

  contracts:
    investigation: "invoice-investigation-v1"
    policy: "policy-assessment-v1"
    orchestration: "invoice-case-v1"

  model:
    provider: "approved-provider"
    modelIdentifier: "approved-model-version"
    credentialReference: "environment-secret-reference"

  environment:
    erpConnection: "ERP_READ"
    policyConnection: "POLICY_READ"
    llmConnection: "LLM_CONNECTION"

  caseState:
    store: "external-case-database"
    schemaVersion: "2"
    oldSchemaReadable: true

  approvals:
    privilegedTransactionEnabled: false

  observability:
    caseCorrelationKey: "caseId"
    monitorOwner: "IntegrationOperations"
    alertOwner: "AIPlatformOperations"

Notice that the manifest records credential references, not credential values. It also records the model identifier and prompt/configuration fingerprints so an apparently identical agent can be compared across releases.

Checkpoint: Every deployable or environment-dependent resource has an owner, version or configuration reference, and a verification method. No required dependency is described merely as "configured normally."

Phase 2 — Establish separate environment configuration

Starting location: The target OIC TEST instance, before importing the project.

Provision and test the target environment's connections, certificates, credentials, identities, and case-store namespace first. After the project is imported in Phase 3, reconcile its connection references with these TEST resources. The imported project is not an assumption or prerequisite for this phase. Do not copy live ERP credentials, production API keys, production customer data, or broad administrator privileges into the test environment.

TEST environment configuration
Configuration TEST value or control
ERP connectionTest endpoint with read-only synthetic-data permissions.
Policy connectionTest policy source or controlled policy fixture.
LLM connectionApproved provider, endpoint and separately managed API credential.
Agent permissionsOnly required tools and target projects.
Case storageSeparate test database or namespace containing synthetic cases.
MonitoringTest log destinations, metrics and notification recipients.

Project-level access must also be reviewed after import. Oracle's project import process permits restricting who can edit, view and monitor imported project resources. Avoid selecting unrestricted access merely to make a deployment easier.

Checkpoint: Every target connection tests successfully, all environment-specific values resolve to TEST resources, and the runtime identities have only the required permissions.

Phase 3 — Promote the OIC project artifacts

Starting location: DEVELOPMENT → Projects → Your project → Deploy.

Oracle Integration 3 supports project deployments containing selected integration versions. Use the explicit versions selected for the release rather than relying on whichever version happens to be latest.

Step 1: Create the deployment in development.

  • Open the project and its Deploy tab.
  • Create a project deployment for the intended release.
  • Select the exact integration versions in the manifest.
  • Review the deployment composition before saving.

Step 2: Export the project deployment.

Use the project export procedure and select the specific project deployment. Export the CAR artifact to a controlled location or configured repository. Record its SHA-256 checksum and associate it with the release commit.

Step 3: Import into the separate test instance.

In the target instance, open Projects, select Add, and import the CAR file. Configure the project's access permissions appropriately. Reconcile environment-specific connections and other dependencies before attempting runtime activation.

Do not manually edit the exported IAR or CAR files to modify connection definitions. Oracle documents restrictions on editing exported integration archives outside OIC.

Step 4: Activate only the supporting and agent-tool integrations needed to build the target agents.

After confirming their target connections, activate and test the read-only evidence and policy integrations that will be registered as agentic AI tools. For each integration used as an agentic AI tool, verify the project membership, Active status, REST Adapter trigger, POST operation, and JSON request and response definitions. Do not activate the case-intake route or CASE_ORCHESTRATOR yet. Keep the coordinator unavailable to business traffic until its target agents and any callback receiver are ready.

The integration activation APIs can retrieve integration activation status and activation errors, which is useful for an external deployment pipeline.

Checkpoint: The expected project and integration versions exist in TEST, target connections are configured, and only the supporting/agent-tool integrations required by the next phase have been activated and tested. The coordinator and intake route remain disabled.

Phase 4 — Re-establish and activate the AI-agent dependencies

This is the most important multi-agent-specific promotion step in the current documented release context.

Oracle's workaround for the project-deployment activation issue is to create a new AI agent and activate it in the target OIC instance. The target agent must reference valid target-side dependencies.

Starting location: TEST → Projects → AP_MULTI_AGENT → AI Agents.

Reconcile the target AI resources in dependency order:

  1. Validate tool integrations. Confirm that each registered agentic AI tool points to the intended imported integration and target connection.
  2. Validate the thinking pattern. Ensure the required pattern exists and is correctly configured for the approved model provider and target LLM connection.
  3. Activate the thinking pattern. Oracle's documented agent-building procedure requires the pattern to be active before it can be selected when creating an agent.
  4. Create the target AI agent. Reproduce the intended agent name, identifier, role, guidelines, thinking-pattern dependency, and tool associations.
  5. Reconcile prompt templates. Compare the actual imported or recreated template text against the source-controlled definition. Recreate templates affected by the pre-26.04 import issue.
  6. Save and activate the agent. Check its target status. Only an Active agent is available for selection in the AI Agent callback configuration.
  7. Configure and activate the callback receiver, when asynchronous tools are used. Build or reconcile the separate integration that receives the external completion event and contains the AI Agent action configured as Callback AI agent. Map the conversation ID, tool-flow ID, tool-invoke-event ID, and response payload as described in Phase 7. Activate and test this receiver only after the target agent is Active.
  8. Keep the coordinator gated. Do not expose the case orchestration integration to new business traffic until both specialist agents, their tools, and any required callback receiver have passed their target-side checks.

A useful distinction exists between the UI and API lifecycle. The Oracle Developer API documents AI-agent creation, retrieval and deletion, and separate operations for patterns, tools and prompt templates. Its published AI-agent endpoint list does not provide a documented equivalent of the AI-agent UI activation action. Do not equate a successful API create request with activation. For this release procedure, use the documented target-side UI activation action and verify the resulting status.

Release decision: What can be automated?

Project archives and documented project/integration REST operations can form part of a deployment pipeline. AI-agent resource APIs also provide selected create, retrieve and delete operations.

However, this does not establish a single automated API operation for promoting, activating and validating every agent, pattern, tool and template. Keep target-side reconciliation and activation as an explicit release gate until the entire lifecycle is verified for the target release.

Checkpoint: Both specialist agents are Active in TEST, their patterns and tools are correct, any required callback receiver is active and mapped, and the coordinator references the intended target agents. The coordinator remains gated until the controlled synthetic-test activation in Phase 5.

Phase 5 — Verify contracts and perform release smoke tests

A deployment can contain the right artifact versions but still fail because the actual response contract changed. Validate both the individual specialists and the whole case workflow.

Before invoking CASE-1001: after the specialist agents and any required callback receiver are Active, activate CASE_ORCHESTRATOR in TEST for controlled synthetic testing. Restrict its reachable caller or intake gate to the test identity and synthetic fixtures. This is not permission to expose the new release to normal business traffic; the production traffic gate remains closed until the release approval step.

For each specialist, verify the contract expected by its consumer:

  • Required identifiers are present and match the current case.
  • Source references and freshness information survive the handoff.
  • Numeric values use the intended representation and currency.
  • Missing or malformed fields produce a controlled validation error.
  • Unsupported agent output cannot be interpreted as successful business completion.

Run the complete CASE-1001 scenario. Check the actual tool integrations, agent statuses, specialist outputs, coordinator validation and final recommendation. Confirm that no write-capable tool was invoked.

Then introduce a boundary test: remove the receipt fixture or mark the receipt quantity as insufficient. The system should identify missing or mismatching evidence and route the case to review, rather than inventing a match.

Finally, disable the policy lookup connection temporarily or simulate an unavailable tool response. The expected result is a controlled incomplete case or manual handoff, not a fabricated policy conclusion. For this tutorial, an application-defined status such as POLICY_SOURCE_UNAVAILABLE may be persisted with a non-terminal case state; do not mark the case complete merely because the agent returned a fluent response.

For each tool contract, record the request sent by the coordinator, the exact response received, the validation outcome, and the case-state transition. The JSON blocks above are example contracts, not proof that the corresponding integrations or agents have been configured in a tenant.

Checkpoint: The release reproduces the expected business result, fails safely on incomplete evidence, and records enough evidence to diagnose an unavailable dependency.

Phase 6 — Define deployment ordering, gates and gradual rollout

A deployment pipeline must respect dependencies. Activating the case orchestration before its tools and specialist agents are available creates an avoidable failure window.

Use this order for the target environment:

1. Target foundation — Connections, credentials, certificates, environment configuration and access permissions.

2. Supporting resources — Libraries, lookups, external case-state dependencies and policy fixtures.

3. Agent-tool integrations — Import, configure, activate and test the read-only integrations that agents will call.

4. Agent patterns — Configure and activate the required thinking patterns and model connections.

5. Specialist agents — Create or reconcile target agents, associate tools/templates, save and activate.

6. Callback receiver and coordinator — Configure and activate any required callback integration after its target agent is Active; then activate the case orchestrator and verify specialist references.

7. Controlled release — Execute smoke tests, receive approval, and allow the intended traffic cohort.

A release gate is a decision that must be satisfied before the next deployment stage begins. Useful gates include successful archive integrity verification, connection tests, agent activation, smoke-test evidence, security approval and business-owner acceptance.

A practical CI/CD design can use the following division of responsibility:

Pipeline stages and responsibilities
Pipeline stage Responsibility
BuildRecord source revision, release manifest and artifact checksum.
PromoteExport/import project and selected integration versions.
ConfigureApply target environment connections and credentials.
Reconcile agentsCreate, configure, activate and verify AI-agent dependencies.
TestRun automated integration checks and agent/business-result tests.
ApproveReceive technical, security and business approval.
ReleaseEnable the authorized traffic cohort and monitor the release.

Oracle Integration's GitHub repository capability supports project and project-deployment export and import, version control, backup and movement between instances. It is a repository collaboration and promotion capability, not proof that the complete deployment pipeline, smoke tests, approvals and release orchestration run automatically inside OIC.

Oracle documents GitHub repository restrictions, including the supported github.com repository type and the use of personal access tokens for that integration. Visual Builder Studio is identified as a separate Oracle CI/CD approach. Select the pipeline technology according to enterprise policy.

Gradual rollout: Do not assume OIC AI agents provide an automatic percentage-based canary or blue-green switch.

Instead, use a controlled upstream API Gateway, calling application, or versioned orchestration entry point to route an explicitly selected cohort to the candidate release. Maintain case-level release affinity so that an existing case continues using a compatible release. When the required routing capability is unavailable, use a small controlled pilot rather than claiming weighted traffic splitting.

Checkpoint: Each promotion stage has a named owner, an approval condition and an observable success signal. The new cohort can be identified separately from existing cases.

Phase 7 — Simulate a faulty release and recover safely

A release rehearsal is incomplete until the team demonstrates what happens when the candidate release behaves incorrectly.

Fault injection: Prepare candidate release 1.4.1 with a deliberately faulty prompt or specialist-output change. For example, make the investigation result omit mandatory evidence-source references.

Run the smoke test and observe the result. The orchestration integration should reject the incomplete specialist result because the required source evidence cannot be validated. The case should not receive a normal final recommendation.

Now exercise containment.

Implement a controlled operational gate in the intake orchestration or its upstream service. For example, an environment-scoped configuration record can expose these settings:

{
  "intakeMode": "MANUAL_ONLY",
  "newCaseProcessingEnabled": false,
  "mutationsEnabled": false,
  "activeRelease": "1.4.0",
  "reason": "Candidate release failed evidence validation"
}

These are tutorial-defined application-level controls, not Oracle-reserved OIC settings.

The controls must perform actual work:

  • The intake gate stops accepting cases into the faulty agent workflow.
  • New submissions are explicitly routed to an approved manual queue or returned with a controlled maintenance response.
  • The mutation gate prevents a downstream transaction integration from performing writes if such an integration exists.
  • Existing cases retain their durable status, evidence and release reference.
  • Operations can identify which cases were accepted before the containment decision.

Enforce a mutation restriction in the actual integration, authorization layer or target application. An agent instruction saying "do not write during maintenance" is not an adequate control.

Protect the in-flight cases.

Before restoring the previous release, identify all cases that have not reached a terminal state. CASE-1001 is the positive smoke-test case shown earlier; it has reached the business outcome WAITING_FOR_REVIEW and is not waiting for a tool callback. The following two additional synthetic records, CASE-1002 and CASE-1003, represent the two asynchronous cases in the opening scenario:

[
  {
    "caseId": "CASE-1002",
    "releaseId": "1.4.0",
    "caseSchemaVersion": 2,
    "status": "WAITING_FOR_TOOL",
    "lastCompletedStep": "INVOICE_EVIDENCE_REQUESTED",
    "pendingCallback": {
      "agentCode": "INVOICE_INVESTIGATION",
      "conversationId": "conversation-example-1002",
      "toolInstanceId": "tool-instance-example-1002",
      "callbackStatus": "AWAITING"
    }
  },
  {
    "caseId": "CASE-1003",
    "releaseId": "1.4.0",
    "caseSchemaVersion": 2,
    "status": "WAITING_FOR_TOOL",
    "lastCompletedStep": "POLICY_SOURCE_REQUESTED",
    "pendingCallback": {
      "agentCode": "POLICY_INTERPRETATION",
      "conversationId": "conversation-example-1003",
      "toolInstanceId": "tool-instance-example-1003",
      "callbackStatus": "AWAITING"
    }
  }
]

These identifiers are examples. In a real run, persist the actual conversation ID and the runtime tool instance ID returned or exposed by the OIC integration metadata. Also persist the owning release, case schema version, tool name, callback status, and last completed orchestration step. Do not use a conversation ID as a substitute for the durable business case record.

Oracle's callback documentation describes the following callback input parameters. The values below illustrate the application contract for CASE-1002; they are not a universal endpoint URL or a complete Oracle REST API request:

{
  "toolFlowId": "tool-instance-example-1002",
  "conversationId": "conversation-example-1002",
  "toolInstanceId": "tool-instance-example-1002",
  "responsePayload": "{\"caseId\":\"CASE-1002\",\"lookupStatus\":\"COMPLETED\",\"evidenceReference\":\"RCV-7721\"}"
}

For the OIC Callback AI agent action, map Conversation Id from the conversation ID obtained from the x-agentic-callback-uri header. Map Tool Flow Id from the runtime tool instance ID. Oracle also requires Tool Invoke Event Id; its documentation says to use the same instance ID for both Tool Flow Id and Tool Invoke Event Id. Map Payload to the stringified JSON response. The callback can be sent only while the associated conversation is in Waiting for tool to process.

Configure the asynchronous tool integration trigger with Oracle's x-agentic-callback-uri header. At runtime, Oracle populates it with the callback URL containing the conversation ID; extract and persist that ID. Read the tool instance ID from Integration Metadata in the integration mapper and persist it with the case. The separate callback integration then invokes the Callback AI agent action. After callback success, verify that the agent resumed and only then advance the application case state. Protect callback handling with an application-level state check so a repeated external completion event does not create duplicate case transitions.

Cases still owned by the prior release must not accidentally resume under an incompatible prompt, contract or state schema. Either keep the old compatible processing path available, complete or manually resolve those cases, or implement a controlled state migration with explicit compatibility tests.

Restore the previous release.

  1. Keep new intake in manual-only mode.
  2. Identify the faulty integration and its exact deployed version.
  3. Restore the previous stable integration version.
  4. Restore the corresponding agent definitions, prompts, pattern configuration and environment references from the release manifest.
  5. Activate and verify the restored components.
  6. Check that existing case records remain readable and that callback references still resolve.
  7. Run the original smoke tests before enabling new intake.
  8. Release the manual queue only after the responsible owners approve recovery.

There is an important OIC-specific rollback consideration. Oracle documents that reverting an entire project deployment can affect every integration in that deployment. In its production rollback guidance, Oracle recommends deactivating only the faulty integration version and reactivating the previous stable version, rather than rolling back the entire set of integrations unnecessarily.

Do not confuse this deployment procedure with reversing a completed invoice transaction. Any future ERP posting requires its own transaction status reconciliation, authorization, duplicate protection and compensating business process.

Checkpoint: The intentionally faulty release is contained, the previous application version is restored, and in-flight case records have not been discarded or silently reinterpreted.

06. Test and Prove the Production Release

The release gate must measure system behavior, not just successful deployment operations or individual agent responses.

Release verification scenarios and evidence
Test case Expected behavior Evidence to inspect
Positive case: CASE-1001 Correct 4.17% price variance and manual-review recommendation. Tool arguments, source references, both agent outputs and final case state.
Incomplete evidence Controlled review or rejection; no invented receipt match. Validation error, evidence completeness and case disposition.
Unavailable specialist tool Incomplete or pending case with controlled manual handling. Integration error, agent activity status and case correlation.
Faulty release Release gate rejects missing evidence and contains new intake. Validation failure, release ID and manual-routing evidence.
Rollback with pending work: CASE-1002 and CASE-1003 Both cases retain their original release/schema ownership and valid callback correlation; each resumes only through a compatible target agent and callback path. Conversation IDs, tool instance IDs, Tool Flow ID and Tool Invoke Event ID mappings, callback status, restored versions and recovery audit.

Execution status for this article: Not executed. These are the prescribed verification scenarios and expected results. No tenant access, release import, activation, fault injection, or rollback evidence was supplied or performed for this tutorial.

The release should be approved only after the actual execution evidence is attached to the release record and reviewed by the relevant owners.

07. Troubleshooting and Recovery

Troubleshooting symptoms and targeted checks
Symptom Likely layer Targeted check and correction
Project imports but agent remains inactive. AI-agent promotion/activation. Apply the documented target-side agent recreation and activation workaround.
Agent cannot invoke its tool. Tool registration, integration version or target connection. Verify integration activation, tool association, request contract and connection test.
Prompt output differs after import. Prompt template compatibility. Compare with the source-controlled prompt and recreate affected pre-26.04 templates.
Asynchronous agent remains waiting. Callback or durable correlation data. Check conversation ID, tool instance ID, Tool Flow ID and Tool Invoke Event ID.
Restored release cannot read an existing case. Persisted-state compatibility. Identify the case schema and release version; restore a compatible reader or perform a tested migration.

Do not repeatedly resubmit a case simply because a previous agent or integration call timed out. First establish whether the operation completed, whether state was committed, and whether a duplicate invocation is safe.

For failed OIC integration instances, the Developer API provides a replay operation. Replay means executing the selected integration instance again; it is not transaction compensation. Before replaying any flow that might invoke a write-capable external system, establish the target transaction's outcome and its idempotency behavior.

08. Enterprise Operating Controls

8.1 Establish correlated observability

OIC integration monitoring and AI-agent activity information answer different operational questions. A specialist's successful response does not establish that the complete case was correctly coordinated.

Build an application-level correlation record containing:

  • Business correlation: case ID, invoice ID and business status.
  • Release correlation: solution release, integration version, prompt/configuration digest and case schema version.
  • Runtime correlation: OIC integration instance ID and each specialist's AI-agent conversation ID.
  • Asynchronous correlation: callback identifier, tool instance ID, expected next step and callback outcome, where applicable.
  • Operational outcome: completion time, manual handoff reason, validation result and responsible support owner.

Oracle documents an AI Agent action called Get AI Agent Activity Stream. It uses the conversation ID to retrieve an agent's activity information, including its current status. This is useful for checking whether a specialist is progressing, waiting or otherwise incomplete.

OIC service-instance monitoring also exposes operational and usage information such as billing message data, processing and invocation times, and adapter requests. OCI metrics and alarms can help monitor instance-level behavior and usage.

Do not assume that combining these facilities automatically produces a full distributed trace across the coordinator, all specialists and external business systems. Maintain your own case-to-runtime correlation record and preserve the relevant identifiers.

8.2 Define service objectives and actionable alerts

A service objective defines the operational behavior the business expects. A threshold determines when support action is required. Neither should be selected simply because a monitoring console offers a convenient default.

For the invoice case workflow, establish measurable objectives such as:

Operational measures and alert conditions
Operational measure What it establishes Alert condition
Case completion rateShare of accepted cases reaching an approved terminal outcome or controlled manual handoff.Rate falls below the agreed business objective.
End-to-end case latencyTime from case acceptance to recommendation or handoff.p95 or maximum duration exceeds the service target.
Pending callback ageTime a case has remained waiting for an asynchronous response.A pending callback exceeds its expected completion window.
Evidence validation failuresFrequency of missing, stale or contradictory evidence.Unexpected increase or a release-specific spike.
Cost per completed caseCombined OIC and LLM operational cost against business value.Sustained cost increase beyond the approved budget.

Use a representative baseline to set actual numerical limits. For example, measure p95 case latency over normal and peak loads before defining an alert threshold. A model's own confidence score must not be treated as a substitute for these business and technical measures.

8.3 Assign support and escalation ownership

A multi-agent solution needs explicit ownership across both integration operations and AI behavior.

  • Integration operations: OIC activation, failed instances, connections, API availability and runtime recovery.
  • AI platform operations: model endpoint availability, configuration drift, prompt changes, agent activity and usage.
  • Application support: ERP evidence, invoice state and application-specific errors.
  • Accounts Payable owner: policy interpretation approval, manual-review outcomes and business acceptance.
  • Security owner: credentials, identity permissions, audit access and privileged-action control.

An operational alert without an owner and a defined action is merely a notification. Every critical alert should identify who investigates it, what evidence they inspect, and which containment action is authorized.

8.4 Retention, audit and recovery testing

  • Standard and Enterprise editions: 32 days of production tracing data by default.
  • Healthcare edition: 184 days of production tracing data.
  • Audit tracing data: 8 days.
  • Debug tracing data: 24 hours.

The ability to change production-trace retention is documented for Enterprise edition; Standard and Healthcare editions do not provide that configurable retention setting in this documentation.

Treat these as OIC tracing-retention values, not as a complete business-record retention policy. Your case store and audit evidence may have different legal, contractual and operational retention requirements.

Avoid storing unnecessary invoice payloads, sensitive evidence or credentials in agent prompts and verbose production logs. Define who can view case evidence, runtime details and audit information, and record administrative changes.

Perform recovery rehearsals that restore the case store from backup and verify that case identifiers, release references, pending approvals, and callback mappings remain usable. A backup is not proven until restoration and consistency checks succeed.

8.5 Review model, policy and cost changes as release changes

A model-provider change, model identifier change, system instruction update or policy change can alter the business result even when the OIC integration versions remain unchanged.

Treat these as versioned changes in the release manifest. Re-run the representative positive, negative and failure datasets before enabling the updated configuration in production.

For costs, review OIC message consumption, processing behavior and adapter usage alongside provider-side LLM usage and billing. An OIC usage increase and a model-token increase represent different cost dimensions; both need to be visible to the service owner.

09. Completion Checklist and Remaining Work

The following checklist describes the evidence required to accept the release rehearsal.

  • Release manifest contains the complete artifact and environment dependency inventory.
  • Source archive, source-control commit and checksums are associated with one release identifier.
  • Development, test and production configuration and credentials are separated.
  • Exact integration versions are selected for the target project deployment.
  • CAR export and import are performed using the documented procedure.
  • Target connections and security configuration have been validated.
  • AI-agent dependencies have been reconciled and the target agents activated.
  • Positive, incomplete-evidence and unavailable-tool tests have recorded results.
  • Faulty-release containment and individual-integration rollback have been rehearsed.
  • In-flight cases and callback references remain consistent.
  • Monitoring ownership, alert thresholds, retention and incident contacts are documented.
  • Business and technical release approvals have been recorded.

Starting artifacts: The source OIC project, its dependent agent and integration definitions, synthetic fixtures and environment configuration references.

Ending artifacts: A versioned release manifest, imported TEST project, reconciled agent definitions, smoke-test evidence, launch checklist and incident runbook.

The target-side recreation of AI agents is a known workaround, not proof of a fully automated agent promotion lifecycle. A production release must be blocked when this reconciliation or any other required dependency is incomplete.

10. Five Useful FAQs

1. Can I export the entire multi-agent system as one CAR file?

A project or project deployment can package supported OIC project resources, but a CAR archive must not be treated as a complete environment backup. Remote shared connections and local connection override configurations are excluded from project export.

2. Does successful project deployment mean every AI agent is active?

No. Project import, integration activation and AI-agent activation are distinct verification points. Apply the documented workaround by creating and activating the target AI agent when affected by the known deployment issue. Verify the actual agent status and invoke it before approving the release.

3. Should I roll back the complete project when one integration is faulty?

Not automatically. Oracle documents that rolling back a complete project deployment can deactivate every integration included in that deployment. For production, Oracle recommends restoring only the affected integration version where possible. Check dependent agent contracts and in-flight cases before making the change.

4. What happens to cases that are waiting during an upgrade?

They must remain associated with a compatible release and case-state schema. Preserve their durable state and any outstanding callback identifiers or approval references. Continue using a compatible older processing path or complete an explicitly tested migration. Do not assume that changing an agent or integration automatically migrates unfinished business work.

5. Does rolling back OIC reverse transactions already completed in ERP?

No. Application rollback restores deployed behavior. A committed ERP transaction requires a separate, authorized business compensation or correction process. Before replaying an integration, reconcile its actual transaction outcome and check duplicate-prevention controls.

11. Additional References

The following Oracle primary sources inspected for this article. Individual documents have their own publication or update dates. The Known Issues page is mutable, so recheck it during each release approval.

  1. Oracle Integration 3 — Agentic AI Issues. November 2026 known issues, AI-agent project-deployment activation, pre-26.04 prompt templates and conversation ID behavior.
  2. Oracle Integration 3 — Export a Project. Project archive behavior, connection exclusions, export location and deployment selection.
  3. Oracle Integration 3 — Create and Manage a Project Deployment. Integration version selection and production rollback guidance.
  4. Oracle Integration 3 — Workflow for Deploying Automation Solutions. Documented development-to-test/production promotion sequence.
  5. Oracle Integration 3 — Create or Import a Project. Project import and access-control options.
  6. Oracle Integration 3 — Integrate Projects and Project Deployments with a GitHub Repository. GitHub integration capabilities and limitations.
  7. Oracle Integration 3 — Developer REST API Reference. Project, deployment, integration and AI-agent resource operations.
  8. Oracle Integration 3 — Create a Deployment in a Project. Project-deployment REST request and supported deployment resources.
  9. Using Agentic AI in Oracle Integration 3 — Steps to Build an AI Agent. AI-agent dependencies, including the required REST trigger, POST operation, and JSON request/response for agentic AI tool integrations.
  10. Oracle Integration 3 — Create the Integration: Auto Approve Expense Report. Tool integration project, activation, REST trigger, and JSON payload requirements.
  11. Oracle Integration 3 — Configure the Agentic AI Thinking Pattern. Pattern configuration and activation prerequisite.
  12. Oracle Integration 3 — Get the Activity Stream for an AI Agent. Retrieving AI-agent activity and status using a conversation ID.
  13. Oracle Integration 3 — Send a Response to an AI Agent with a Callback. Asynchronous tool completion, callback identifiers and mapping requirements.
  14. Oracle Integration 3 — Edit the Data Retention Period. Production, audit and debug trace retention behavior.
  15. Oracle Integration 3 — Component: Integrations. Runtime limits and operational considerations.
  16. Oracle Integration 3 — Stay Within Budget: Track Billing Metrics. Monitoring of billing messages, processing and invocation times, and adapter requests.

12. Finally — Summary

A production-ready multi-agent system is a versioned, governed business application, not simply a collection of activated agents. This release-engineering playbook has examined complete release inventory, environment separation, OIC project promotion, target-side agent activation, contract compatibility, controlled rollout, and recovery of unfinished cases.

The most important engineering lesson is that deployment success and business continuity are different acceptance criteria. Preserve case state, correlate every specialist execution, validate the actual business outcome, and keep rollback separate from transaction compensation. Production confidence comes from reproducible releases and rehearsed recovery, not from a green activation status alone.

Comments