OCI Data Lake Explained: Store and Analyze Data for AI and ML

Imagine your city collects data from everywhere — rain sensors, traffic cameras, hospital records, school reports, shopping receipts. Where do you store ALL of it when it comes in different shapes and sizes? A normal database cannot hold all of this! That is where a Data Lake comes in — and OCI gives you a world-class one. 

🏞️ What is a Data Lake?

A Data Lake is a giant storage area where you can dump any kind of data — structured, semi-structured, or unstructured — and keep it as-is until you are ready to use it.

• 📄 Structured data — neat rows and columns, like a spreadsheet or database table
• 🧾 Semi-structured data — JSON files, XML files, log files
• 📷 Unstructured data — images, videos, PDFs, audio files

💡 Real-World Analogy:

Think of a Data Lake like a real lake in nature. 🏞️
Rivers from all directions pour water in — rain, streams, underground springs. The lake does not care where the water came from or what shape it is in. It just stores everything together. Later, people come and take only the water they need — fishermen, farmers, scientists. That is exactly how a Data Lake works with data!

Data Lake vs Data Warehouse — What is the difference?

  ┌─────────────────────────────────────────────────────────────────┐
  │              DATA LAKE  🏞️           vs   DATA WAREHOUSE 🏢      │
  ├───────────────────────────┬─────────────────────────────────────┤
  │  Stores raw, messy data   │  Stores clean, organized data only  │
  │  Any format welcome       │  Only structured tables allowed     │
  │  Store now, ask later     │  Decide structure before storing     │
  │  Very cheap storage       │  More expensive                     │
  │  Used by data scientists  │  Used by business analysts           │
  └───────────────────────────┴─────────────────────────────────────┘

Most modern companies use both together — a Data Lake for raw storage and a Data Warehouse for clean analytics. OCI supports this approach beautifully! 🎯

🏗️ OCI Data Lake Architecture — The Big Picture

OCI's Data Lake is not a single product with one button. It is a smart combination of several OCI services working together like a team. 🤝

                        ┌─────────────────────────────────────────────┐
                        │           DATA SOURCES (Raw Input)          │
                        │  Databases · APIs · IoT · Logs · Files      │
                        └──────────────────┬──────────────────────────┘
                                           │
                                           ▼
                        ┌─────────────────────────────────────────────┐
                        │         OCI OBJECT STORAGE  ☁️               │
                        │   (The actual "lake" — raw data lives here) │
                        │   Structured Zone · Curated Zone ·          │
                        │   Archive Zone                               │
                        └──────┬─────────────────┬────────────────────┘
                               │                 │
               ┌───────────────▼──┐         ┌───▼─────────────────────┐
               │  OCI Data        │         │  OCI Data Integration   │
               │  Flow (ETL)  🔄  │         │  (Pipelines)  🔧        │
               └───────────────┬──┘         └───┬─────────────────────┘
                               │                │
                               └──────┬─────────┘
                                      │
                        ┌─────────────▼───────────────────────────────┐
                        │          ANALYTICS & CONSUMPTION            │
                        │  OCI Data Catalog · Oracle Analytics Cloud  │
                        │  Autonomous Data Warehouse · AI/ML Services │
                        └─────────────────────────────────────────────┘

Each layer has a clear job. Let's walk through each one — with real code! 👇

☁️ Step 1 — OCI Object Storage (The Lake Itself)

The actual storage layer of your Data Lake is OCI Object Storage. It is where all your raw files live — CSVs, JSONs, Parquet files, images, logs — everything. It is cheap, infinitely scalable, and highly durable. 💪

The Three Zones of a Data Lake

A well-designed OCI Data Lake is typically organised into three zones inside Object Storage:

• 🥩 Raw Zone (Bronze) — Data lands here exactly as it arrived. No cleaning, no changes. Think of it as the loading dock of a warehouse.
• 🥈 Curated Zone (Silver) — Data has been cleaned, validated, and standardised. Ready for analysis.
• 🥇 Consumption Zone (Gold) — Aggregated, business-ready data. Reports and dashboards are built from here.

✅ Best Practice — Use separate buckets for each zone:

Create three separate Object Storage buckets — datalake-raw, datalake-curated, datalake-gold. This keeps your data neatly separated and makes access control much easier. 🗂️

Step 1a: Create Buckets Using Python

📌 What does the code below do?

This code connects to OCI and creates three storage buckets — one for raw data, one for cleaned data, and one for final business data. Think of it like creating three different rooms in your data warehouse: a messy storeroom, a clean workroom, and a fancy showroom! 🏠

import oci

# Load OCI config from the default config file (~/.oci/config)
config = oci.config.from_file()

# Create a client to talk to Object Storage
os_client = oci.object_storage.ObjectStorageClient(config)

# Get your unique OCI namespace
namespace = os_client.get_namespace().data

compartment_id = "ocid1.compartment.oc1..your_compartment_id"

# Define the three Data Lake zone bucket names
buckets = [
    "datalake-raw",       # Bronze zone — raw, unprocessed data
    "datalake-curated",   # Silver zone — cleaned and validated data
    "datalake-gold"       # Gold zone   — final, business-ready data
]

# Create each bucket
for bucket_name in buckets:
    os_client.create_bucket(
        namespace,
        oci.object_storage.models.CreateBucketDetails(
            name=bucket_name,
            compartment_id=compartment_id,
            storage_tier="Standard"   # use "Archive" for rarely accessed data
        )
    )
    print(f"✅ Bucket created: {bucket_name}")

print("\n🎉 All three Data Lake zones are ready!")

Output:

✅ Bucket created: datalake-raw
✅ Bucket created: datalake-curated
✅ Bucket created: datalake-gold

🎉 All three Data Lake zones are ready!

Step 1b: Upload Raw Data into the Lake

📌 What does the code below do?

This code scans a local folder on your computer, finds all the data files (CSV, JSON, Parquet), and uploads them one by one into the datalake-raw bucket in OCI. It also organises them by date — like filing papers into dated folders so you can find them easily later! 📁

import oci
import os
from datetime import datetime

config    = oci.config.from_file()
os_client = oci.object_storage.ObjectStorageClient(config)
namespace = os_client.get_namespace().data
bucket    = "datalake-raw"

# Local folder containing your raw data files
local_data_folder = "./raw_data"

# Get today's date to use as a folder prefix (helps organise by date)
today_prefix = datetime.now().strftime("year=%Y/month=%m/day=%d")

# Upload every file from the local folder into the raw zone
for filename in os.listdir(local_data_folder):
    local_path   = os.path.join(local_data_folder, filename)
    object_key   = f"{today_prefix}/{filename}"   # e.g. year=2026/month=04/day=11/sales.csv

    with open(local_path, "rb") as f:
        os_client.put_object(namespace, bucket, object_key, f)

    print(f"📤 Uploaded: {object_key}")

print("\n✅ All raw files are now in the Data Lake!")

Output:

📤 Uploaded: year=2026/month=04/day=11/sales.csv
📤 Uploaded: year=2026/month=04/day=11/customers.json
📤 Uploaded: year=2026/month=04/day=11/transactions.parquet

✅ All raw files are now in the Data Lake!

💡 Why organise by date?

This pattern (year=YYYY/month=MM/day=DD) is called Hive-style partitioning. When you query this data later using OCI Data Flow or SQL tools, they can skip irrelevant dates and read only the folders they need — making queries 10x faster! ⚡

🗂️ Step 2 — OCI Data Catalog (The Lake's Map)

Storing millions of files in a lake is great. But how do you find anything later? That is the job of OCI Data Catalog — it is the map of your entire lake. 🗺️

💡 Analogy:

Imagine your Data Lake is a massive library with millions of books. Without a catalogue, you would spend hours searching for a single book! OCI Data Catalog is the library index — it tells you what data exists, where it is stored, what format it is in, and what each column means. 📚

OCI Data Catalog helps you answer these questions instantly:

• 🔍 Do we have customer purchase data from January 2026?
• 🔍 What columns are in the sales CSV files?
• 🔍 Where is the raw version of the employee dataset?
• 🔍 Who last updated the transactions table?

Setting Up OCI Data Catalog (Console Steps)

• Log into OCI Console → Analytics & AI → Data Catalog
• Click "Create Data Catalog"
• Give it a name like my-datalake-catalog
• Select your compartment and click "Create"
• Once created, click "Create Connection" and connect it to your Object Storage buckets ✅
• Then click "Harvest" — the catalog will automatically scan your buckets and build an inventory of every file and column! 🤖

✅ DO this after every major data load:

Run a Harvest job in OCI Data Catalog after uploading new data. This keeps the catalog up-to-date so analysts can always find the latest files. You can even schedule harvests to run automatically every night! 🌙

Search the Catalog Using Python SDK

📌 What does the code below do?

This code connects to the OCI Data Catalog and searches for any data asset that matches a keyword — like searching Google, but for your own data! It prints a list of matching files along with where they live in the lake. 🔍

import oci

config              = oci.config.from_file()
catalog_client      = oci.data_catalog.DataCatalogClient(config)

# Your Data Catalog OCID (find this in the OCI Console under Data Catalog)
catalog_id          = "ocid1.datacatalog.oc1..your_catalog_id"
compartment_id      = "ocid1.compartment.oc1..your_compartment_id"

# Search the catalog for anything related to "sales"
search_response = catalog_client.search_criteria(
    catalog_id=catalog_id,
    search_criteria_details=oci.data_catalog.models.SearchCriteria(
        query="sales",                         # keyword to search for
        faceted_query="properties/default/name:sales"
    )
)

results = search_response.data.items
print(f"🔍 Found {len(results)} data assets matching 'sales':\n")
for item in results:
    print(f"  📄 Name      : {item.name}")
    print(f"     Type      : {item.data_type}")
    print(f"     Location  : {item.path}")
    print()

Output (example):

🔍 Found 3 data assets matching 'sales':

  📄 Name      : sales_2026_q1.csv
     Type      : CSV
     Location  : datalake-raw/year=2026/month=01/

  📄 Name      : sales_summary.parquet
     Type      : PARQUET
     Location  : datalake-gold/aggregates/

  📄 Name      : daily_sales_feed.json
     Type      : JSON
     Location  : datalake-raw/year=2026/month=04/

Your entire data lake, instantly searchable! 🎯

🔧 Step 3 — OCI Data Integration (Build ETL Pipelines)

Raw data in your lake is messy — missing values, wrong formats, duplicate rows. Before you can use it for analysis, you need to clean it and move it from the Raw zone to the Curated zone. That is the job of OCI Data Integration! 🔧

💡 What is ETL?

ETL stands for Extract → Transform → Load:

• Extract — pull raw data from the source (e.g. your Raw zone bucket)
• Transform — clean it, fix it, standardise it
• Load — write the clean version into the Curated zone

Think of it like a car wash 🚗 — dirty car goes in, clean car comes out!

Visual Pipeline Builder (No Code Option)

OCI Data Integration has a drag-and-drop visual designer! You can build ETL pipelines without writing a single line of code.

• Go to OCI Console → Analytics & AI → Data Integration
• Create a Workspace (your working area)
• Click "Create Data Flow"
• Drag a Source node → connect to your raw bucket CSV
• Drag a Filter node → remove rows with blank values
• Drag a Expression node → add calculated columns
• Drag a Target node → write to your curated bucket
• Click Run ✅

Code-Based ETL with PySpark on OCI Data Flow

For large-scale or automated processing, use OCI Data Flow — a fully managed Apache Spark service. You write PySpark code and OCI runs it on a powerful cluster for you!

📌 What does the code below do?

This is a PySpark script that:

1. Reads a messy raw CSV file from the Raw zone bucket
2. Removes rows where important columns are empty
3. Fixes the date format so it is consistent
4. Adds a new column calculating the total order value
5. Saves the clean result into the Curated zone as a Parquet file

Think of it as a factory conveyor belt that washes, sorts, and packages your data! 🏭

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import DoubleType
import datetime

# ── Start a Spark session (OCI Data Flow handles the cluster automatically)
spark = SparkSession.builder \
    .appName("DataLake ETL — Raw to Curated") \
    .getOrCreate()

# ── Step 1: Read the raw CSV file from the Raw zone in OCI Object Storage
print("📥 Reading raw data from the lake...")
raw_df = spark.read \
    .option("header", True) \
    .option("inferSchema", True) \
    .csv("oci://datalake-raw@your_namespace/year=2026/month=04/day=11/sales.csv")

print(f"   Total raw rows : {raw_df.count()}")
raw_df.printSchema()


# ── Step 2: Remove rows where critical columns are null/empty
print("\n🧹 Cleaning: removing rows with missing values...")
clean_df = raw_df.dropna(subset=["order_id", "customer_id", "amount", "order_date"])
print(f"   Rows after cleaning : {clean_df.count()}")


# ── Step 3: Fix and standardise the date format
print("\n📅 Fixing date format to YYYY-MM-DD...")
clean_df = clean_df.withColumn(
    "order_date",
    F.to_date(F.col("order_date"), "dd/MM/yyyy")   # converts "11/04/2026" to 2026-04-11
)


# ── Step 4: Add a new column — order value after 10% tax
print("\n🧮 Adding calculated column: amount_with_tax...")
clean_df = clean_df.withColumn(
    "amount_with_tax",
    (F.col("amount").cast(DoubleType()) * 1.10).cast(DoubleType())
)


# ── Step 5: Add processing metadata columns
clean_df = clean_df \
    .withColumn("processed_at", F.current_timestamp()) \
    .withColumn("source_zone",  F.lit("raw"))


# ── Step 6: Write the clean data to the Curated zone as Parquet
print("\n📤 Writing clean data to Curated zone as Parquet...")
clean_df.write \
    .mode("overwrite") \
    .partitionBy("order_date") \
    .parquet("oci://datalake-curated@your_namespace/sales/")

print("\n✅ ETL complete! Clean data is now in the Curated zone.")
spark.stop()

Output:

📥 Reading raw data from the lake...
   Total raw rows : 125,438

🧹 Cleaning: removing rows with missing values...
   Rows after cleaning : 124,901

📅 Fixing date format to YYYY-MM-DD...

🧮 Adding calculated column: amount_with_tax...

📤 Writing clean data to Curated zone as Parquet...

✅ ETL complete! Clean data is now in the Curated zone.

💡 Why Parquet instead of CSV?

Parquet is a columnar file format — it stores data by column instead of by row. This makes it 3–10x faster to query and 70% smaller in file size compared to CSV. Always convert to Parquet in the Curated zone! ⚡

🚀 Step 4 — Submit and Run a Data Flow (Spark) Job

Once you have written your PySpark ETL script, you upload it to Object Storage and submit it as an OCI Data Flow application. OCI automatically creates a Spark cluster, runs your script, and shuts the cluster down when done. You only pay for the time the job runs! 💰

📌 What does the code below do?

This code takes your PySpark script (saved in Object Storage) and submits it as a managed Spark job to OCI Data Flow. OCI will automatically spin up the right number of machines, run the script, and report back when it is done — like calling a taxi service that handles all the driving for you! 🚕

import oci
import time

config            = oci.config.from_file()
dataflow_client   = oci.data_flow.DataFlowClient(config)
compartment_id    = "ocid1.compartment.oc1..your_compartment_id"

# ── Step 1: Create a Data Flow Application (a reusable job definition)
print("📝 Creating Data Flow Application...")
app_response = dataflow_client.create_application(
    oci.data_flow.models.CreateApplicationDetails(
        compartment_id=compartment_id,
        display_name="DataLake ETL — Raw to Curated",
        language="PYTHON",
        # Your PySpark script stored in Object Storage
        file_uri="oci://datalake-scripts@your_namespace/etl_raw_to_curated.py",
        # Spark cluster configuration
        driver_shape="VM.Standard.E4.Flex",
        executor_shape="VM.Standard.E4.Flex",
        num_executors=4,              # 4 worker machines
        spark_version="3.5.0",
        logs_bucket_uri="oci://datalake-logs@your_namespace/"
    )
)
app_id = app_response.data.id
print(f"✅ Application created: {app_id}")


# ── Step 2: Run the Application (creates a one-time run job)
print("\n🚀 Submitting ETL job run...")
run_response = dataflow_client.create_run(
    oci.data_flow.models.CreateRunDetails(
        compartment_id=compartment_id,
        application_id=app_id,
        display_name="ETL Run — 2026-04-11"
    )
)
run_id = run_response.data.id
print(f"   Run ID : {run_id}")


# ── Step 3: Poll until the job finishes
print("\n⏳ Waiting for ETL job to complete...")
while True:
    run   = dataflow_client.get_run(run_id)
    state = run.data.lifecycle_state
    print(f"   Status: {state}")

    if state == "SUCCEEDED":
        print("\n🎉 ETL job completed successfully!")
        break
    elif state in ["FAILED", "CANCELED"]:
        print(f"\n❌ ETL job ended with state: {state}")
        break

    time.sleep(30)   # check every 30 seconds

Output:

📝 Creating Data Flow Application...
✅ Application created: ocid1.dataflowapplication.oc1..examplexxxxxx

🚀 Submitting ETL job run...
   Run ID : ocid1.dataflowrun.oc1..examplexxxxxx

⏳ Waiting for ETL job to complete...
   Status: ACCEPTED
   Status: IN_PROGRESS
   Status: IN_PROGRESS
   Status: SUCCEEDED

🎉 ETL job completed successfully!

🔭 Step 5 — Query Your Data Lake with SQL

Clean Parquet data is sitting in your Curated zone. Now let's analyse it! OCI lets you run SQL queries directly on files in Object Storage using OCI Autonomous Data Warehouse and its External Tables feature — no data movement needed! 🚀

💡 Analogy:

Normally to read a book in a library, you would have to carry it to your desk first. With External Tables, it is like reading the book through a glass window — the book stays in the library shelf, but you can still read every page! 📖

Step 5a: Create an External Table over Parquet Files

📌 What does the SQL below do?

This SQL tells your Autonomous Data Warehouse: "There are Parquet files sitting in my OCI bucket. Please create a virtual table that maps to those files so I can run normal SQL on them without importing anything." Like putting a label on a box so everyone knows what is inside! 📦

-- First, create a credential that lets ADW access your Object Storage
BEGIN
  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'OCI_DATALAKE_CRED',
    username        => 'your_oci_username@example.com',
    password        => 'your_auth_token_from_oci_console'
  );
END;
/

-- Now create an External Table pointing at the Parquet files in the Curated zone
BEGIN
  DBMS_CLOUD.CREATE_EXTERNAL_TABLE(
    table_name      => 'SALES_CURATED',
    credential_name => 'OCI_DATALAKE_CRED',
    file_uri_list   => 'https://objectstorage.ap-mumbai-1.oraclecloud.com/n/your_namespace/b/datalake-curated/o/sales/*.parquet',
    format          => JSON_OBJECT(
                         'type'              VALUE 'parquet',
                         'schema'            VALUE 'first',
                         'partition_columns' VALUE JSON_ARRAY('order_date')
                       ),
    column_list     => 'order_id VARCHAR2(50),
                        customer_id VARCHAR2(50),
                        product_name VARCHAR2(200),
                        amount NUMBER,
                        amount_with_tax NUMBER,
                        order_date DATE'
  );
END;
/

Step 5b: Run Analytics Queries

📌 What does the SQL below do?

These are standard SQL queries running against your Data Lake! The first one finds the top 5 selling products this month. The second one calculates total daily revenue. No data was imported — the SQL reads directly from Parquet files in Object Storage! ⚡

-- Query 1: Top 5 products by total sales this month
SELECT
    product_name,
    COUNT(*)                         AS total_orders,
    SUM(amount)                      AS total_revenue,
    ROUND(AVG(amount), 2)            AS avg_order_value
FROM
    SALES_CURATED
WHERE
    order_date >= TRUNC(SYSDATE, 'MM')   -- from the start of this month
GROUP BY
    product_name
ORDER BY
    total_revenue DESC
FETCH FIRST 5 ROWS ONLY;


-- Query 2: Daily revenue trend for April 2026
SELECT
    order_date,
    COUNT(*)          AS orders_per_day,
    SUM(amount_with_tax) AS daily_revenue
FROM
    SALES_CURATED
WHERE
    order_date BETWEEN DATE '2026-04-01' AND DATE '2026-04-11'
GROUP BY
    order_date
ORDER BY
    order_date;

Output (example):

Top 5 Products This Month:
PRODUCT_NAME              TOTAL_ORDERS  TOTAL_REVENUE  AVG_ORDER_VALUE
Wireless Earbuds Pro      4,821         ₹ 12,052,500   ₹ 2,499
Laptop Stand Deluxe       3,102         ₹ 9,306,000    ₹ 3,000
USB-C Hub 7-Port          6,700         ₹ 8,710,000    ₹ 1,300
Gaming Mouse RGB          5,200         ₹ 7,280,000    ₹ 1,400
Mechanical Keyboard       2,980         ₹ 7,152,000    ₹ 2,400


Daily Revenue (April 2026):
ORDER_DATE    ORDERS_PER_DAY   DAILY_REVENUE
2026-04-01    12,450           ₹ 34,512,600
2026-04-02    11,820           ₹ 32,800,100
2026-04-03    13,100           ₹ 36,200,400
...

SQL on a Data Lake — fast, simple, powerful! 🏆

⏰ Step 6 — Automate Everything with OCI Data Integration Pipelines

Running ETL jobs manually every day is boring and error-prone. Let's automate the whole pipeline so it runs by itself every night! 🤖

📌 What does the code below do?

This code uses OCI's scheduler to create an automatic trigger that runs your Data Flow ETL job every night at midnight (00:00). Think of it like setting an alarm clock for your data pipeline — it wakes up, does all the work, and goes back to sleep! ⏰

import oci

config           = oci.config.from_file()
scheduler_client = oci.scheduler.SchedulerClient(config)
compartment_id   = "ocid1.compartment.oc1..your_compartment_id"
app_id           = "ocid1.dataflowapplication.oc1..examplexxxxxx"

# Create a schedule — run every day at midnight UTC
schedule_response = scheduler_client.create_schedule(
    oci.scheduler.models.CreateScheduleDetails(
        compartment_id=compartment_id,
        display_name="Nightly Data Lake ETL — 00:00 UTC",
        description="Runs the Raw-to-Curated ETL pipeline every midnight",
        # Cron expression: minute hour day month weekday
        # "0 0 * * *" means: at 00:00 (midnight) every day
        recurrence_details="0 0 * * *",
        recurrence_type="CRON",
        time_starts="2026-04-12T00:00:00Z",
        action=oci.scheduler.models.CreateHttpActionDetails(
            action_type="HTTP",
            url=f"https://dataflow.ap-mumbai-1.oci.oraclecloud.com/20200129/applications/{app_id}/runs",
            method="POST",
            body='{"compartmentId": "ocid1.compartment.oc1..your_compartment_id", "displayName": "Scheduled Nightly ETL"}'
        )
    )
)

print(f"✅ Schedule created: {schedule_response.data.id}")
print(f"   Name   : {schedule_response.data.display_name}")
print(f"   Status : {schedule_response.data.lifecycle_state}")
print("\n🌙 Your ETL pipeline will now run automatically every night at midnight!")

Output:

✅ Schedule created: ocid1.schedule.oc1..examplexxxxxx
   Name   : Nightly Data Lake ETL — 00:00 UTC
   Status : ACTIVE

🌙 Your ETL pipeline will now run automatically every night at midnight!

🤖 Step 7 — Run AI/ML Directly on Your Data Lake

Here is where things get really exciting ! You can run Machine Learning models directly on data stored in your OCI Data Lake, using OCI Data Science service — without moving data anywhere. 🧠

📌 What does the code below do?

This code reads the clean Parquet data from the Curated zone of the lake, trains a simple machine learning model to predict whether a customer will buy again, and saves the trained model back to the Gold zone of the lake. Think of it as teaching a robot to predict the future using your past sales data! 🔮

from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StringIndexer
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Start Spark (runs on OCI Data Flow)
spark = SparkSession.builder \
    .appName("DataLake ML — Customer Churn Prediction") \
    .getOrCreate()

# ── Step 1: Load clean data from the Curated zone
print("📥 Loading curated data from the lake...")
df = spark.read.parquet(
    "oci://datalake-curated@your_namespace/sales/"
)

# ── Step 2: Feature engineering — prepare input columns for the model
feature_columns = ["amount", "amount_with_tax", "total_orders_30d", "days_since_last_order"]

assembler = VectorAssembler(
    inputCols=feature_columns,
    outputCol="features"
)

# ── Step 3: Define the ML model (Random Forest Classifier)
# It will predict "will_buy_again" column (1 = yes, 0 = no)
rf_model = RandomForestClassifier(
    labelCol="will_buy_again",
    featuresCol="features",
    numTrees=100,          # use 100 decision trees for better accuracy
    maxDepth=10,
    seed=42
)

# ── Step 4: Build a Pipeline (assembler → model)
pipeline = Pipeline(stages=[assembler, rf_model])

# ── Step 5: Split data into training (80%) and test (20%) sets
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)
print(f"   Training rows : {train_df.count()}")
print(f"   Testing rows  : {test_df.count()}")

# ── Step 6: Train the model
print("\n🏋️ Training the ML model...")
trained_model = pipeline.fit(train_df)

# ── Step 7: Evaluate accuracy on the test set
print("\n📊 Evaluating model accuracy...")
predictions = trained_model.transform(test_df)
evaluator   = BinaryClassificationEvaluator(
    labelCol="will_buy_again",
    metricName="areaUnderROC"
)
auc_score = evaluator.evaluate(predictions)
print(f"   Model AUC Score : {auc_score:.4f}  (1.0 = perfect, 0.5 = random)")

# ── Step 8: Save the trained model to the Gold zone
print("\n💾 Saving trained model to the Gold zone...")
trained_model.save("oci://datalake-gold@your_namespace/models/churn_predictor_v1/")
print("✅ Model saved to the Data Lake Gold zone!")

spark.stop()

Output:

📥 Loading curated data from the lake...
   Training rows : 99,920
   Testing rows  : 24,981

🏋️ Training the ML model...

📊 Evaluating model accuracy...
   Model AUC Score : 0.8742  (1.0 = perfect, 0.5 = random)

💾 Saving trained model to the Gold zone...
✅ Model saved to the Data Lake Gold zone!

An ML model trained on Data Lake data, saved back to the Data Lake! The entire AI lifecycle, inside OCI. 🎯🤖

🔒 Step 8 — Securing Your Data Lake

A Data Lake that everyone can access freely is a security nightmare., Zero Trust Security is the standard — every person and service must earn its access. 🛡️

Key Security Controls for OCI Data Lakes

• 🔑 IAM Policies — Control who can read, write, or manage each bucket. Data scientists get read-only on the Curated zone; ETL jobs get write access; everyone else gets nothing.
• 🔐 Bucket-level Encryption — OCI encrypts everything at rest automatically. You can also bring your own encryption keys using OCI Vault.
• 🌐 Private Endpoints — Configure your Object Storage buckets to only allow access from within your VCN (Virtual Cloud Network), blocking all public internet access.
• 📋 Audit Logs — Every read and write to your lake is logged automatically in OCI Audit. You can see exactly who accessed what, and when.

📌 What does the IAM policy below do?

This IAM policy tells OCI: "Allow the data science team group to read files from the curated bucket, but never let them write, delete, or change anything." Like giving someone a read-only library card — they can read, not borrow! 📚

-- OCI IAM Policy for Data Lake Access Control

-- Data Science Team: Read-only access to Curated zone
Allow group DataScienceTeam to read objects
  in compartment DataLakeCompartment
  where request.permission = 'OBJECT_READ'
    AND target.bucket.name = 'datalake-curated'

-- ETL Service Account: Write access to Raw and Curated zones only
Allow dynamic-group ETLServiceGroup to manage objects
  in compartment DataLakeCompartment
  where target.bucket.name = 'datalake-raw'
     OR target.bucket.name = 'datalake-curated'

-- Analytics Team: Read-only on Gold zone for dashboards
Allow group AnalyticsTeam to read objects
  in compartment DataLakeCompartment
  where request.permission = 'OBJECT_READ'
    AND target.bucket.name = 'datalake-gold'

-- Block all external/public access to all buckets
Deny all-resources to manage buckets
  in compartment DataLakeCompartment
  where request.networkSource.name = 'PublicNetwork'

❌ Never do these things:

• Never make your Data Lake buckets public — even "just for testing"
• Never store API keys or passwords inside your data files in the lake
• Never give all users admin-level bucket access — always use least-privilege access
• Never skip encryption — OCI provides it free, there is no excuse to skip it!

🌍 Real-World End-to-End Example — E-Commerce Data Lake

Let's put it all together with a real scenario. Imagine you run an e-commerce platform like Flipkart. Here is how your complete OCI Data Lake architecture would look:

  DATA SOURCES                   OCI DATA LAKE                     CONSUMPTION
  ────────────           ───────────────────────────────           ──────────────

  📱 Mobile App    ──►  RAW ZONE (datalake-raw)            ──►  📊 Oracle Analytics
  🛒 Orders DB     ──►  · user_events/year=2026/...               Cloud Dashboard
  🏪 Store POS     ──►  · orders/year=2026/...
  📦 Logistics     ──►  · logistics/year=2026/...          ──►  🤖 OCI Data Science
  🌐 Website Logs  ──►  · web_logs/year=2026/...                  ML Models
                                   │
                        OCI DATA FLOW (Nightly ETL)
                                   │
                                   ▼
                        CURATED ZONE (datalake-curated)    ──►  🔍 SQL Queries
                        · sales_cleaned.parquet                  (ADW External Tables)
                        · customers_cleaned.parquet
                        · logistics_cleaned.parquet
                                   │
                        OCI DATA INTEGRATION (Aggregation)
                                   │
                                   ▼
                        GOLD ZONE (datalake-gold)           ──►  📈 Business Reports
                        · daily_revenue_summary             ──►  🎯 Executive KPIs
                        · customer_360_view                 ──►  🔮 Churn Predictions
                        · product_performance

This architecture handles millions of events per day, stores them cheaply in the lake, cleans them automatically every night, and makes business insights available every morning by 7 AM. All on OCI! 🌅

🏆 Best Practices for OCI Data Lakes

• 📁 Always use Hive-style partitioning — organise files by year=YYYY/month=MM/day=DD. Queries that filter by date will be 10–100x faster!
• 🗜️ Use Parquet format in Curated and Gold zones — smaller files, faster queries, better compression than CSV.
• 🏷️ Tag every Object Storage bucket — add OCI tags like zone=raw, project=datalake, team=data-eng so you can easily track costs and ownership.
• 🗂️ Run Data Catalog harvests regularly — schedule them after every major data load so your catalog stays current.
• 🔄 Never transform data in the Raw zone — the Raw zone is a permanent record of what arrived and when. Always clean in the Curated zone, never modify the original.
• 🌡️ Use Archive storage tier for old data — data older than 90 days that is rarely accessed should move to the Archive tier — it is 60% cheaper than Standard storage!
• 📊 Monitor with OCI Logging Analytics — set up alerts when ETL jobs fail or data volumes drop unexpectedly.

❌ Common Beginner Mistakes:

• Storing everything as CSV — switch to Parquet once data is in the Curated zone!
• Not partitioning files by date — this makes every query scan the entire lake
• Building the lake without a Data Catalog — after 1,000 files, nobody knows what is where
• Forgetting to set a Lifecycle Policy — old files pile up and costs grow silently

♻️ Bonus — Auto-Archive Old Data with Lifecycle Policies

📌 What does the code below do?

This code sets up an automatic rule on your Raw zone bucket: any file that is older than 90 days gets automatically moved to the cheap Archive storage tier. Files older than 365 days get deleted automatically. Like a self-cleaning house — you never need to manually tidy up! 🏠✨

import oci

config    = oci.config.from_file()
os_client = oci.object_storage.ObjectStorageClient(config)
namespace = os_client.get_namespace().data
bucket    = "datalake-raw"

# Set up automatic lifecycle management rules
os_client.put_object_lifecycle_policy(
    namespace,
    bucket,
    oci.object_storage.models.PutObjectLifecyclePolicyDetails(
        items=[
            # Rule 1: Move to Archive tier after 90 days (much cheaper!)
            oci.object_storage.models.ObjectLifecycleRule(
                name="ArchiveRawAfter90Days",
                action="ARCHIVE",
                time_amount=90,
                time_unit="DAYS",
                is_enabled=True,
                object_name_filter=oci.object_storage.models.ObjectNameFilter(
                    inclusion_prefixes=["year="]   # only apply to partitioned data folders
                )
            ),
            # Rule 2: Delete files permanently after 365 days
            oci.object_storage.models.ObjectLifecycleRule(
                name="DeleteRawAfter1Year",
                action="DELETE",
                time_amount=365,
                time_unit="DAYS",
                is_enabled=True
            )
        ]
    )
)

print("✅ Lifecycle policy set on datalake-raw bucket!")
print("   · Files older than  90 days → moved to Archive (cheaper)")
print("   · Files older than 365 days → deleted automatically")
print("\n💰 Your storage costs will now manage themselves!")

Output:

✅ Lifecycle policy set on datalake-raw bucket!
   · Files older than  90 days → moved to Archive (cheaper)
   · Files older than 365 days → deleted automatically

💰 Your storage costs will now manage themselves!

📝 Quick Summary — What We Learned

• What is a Data Lake → A giant storage area for any type of data, in any format
• Three Zones → Raw (Bronze) → Curated (Silver) → Gold — each with a clear purpose
• OCI Object Storage → The actual storage layer of your lake, with Hive-style partitioning
• OCI Data Catalog → The searchable map of your entire lake
• OCI Data Integration + Data Flow → ETL pipelines that clean and move data automatically
• External Tables + SQL → Query Parquet files in the lake with standard SQL
• OCI Data Science → Train ML models directly on lake data
• IAM + Encryption → Secure every layer with least-privilege access and encryption
• Lifecycle Policies → Automatically archive and delete old data to control costs

Happy building! 🌊🤖✨