From Oracle to MotherDuck using Azure Datalake Gen2

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_oracle_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id adls_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target motherduck_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - Oracle Database

Oracle Database is the world's most widely used enterprise database. LakeXpress uses FastBCP with optimized Oracle connectors for high-performance data extraction to Parquet format.

Features:

•Native Oracle ODP.NET driver via FastBCP
•Full support for Oracle-specific types
•Parallel extraction to Parquet files
•Optimized for large volumes

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Azure Data Lake Storage Gen2

ADLS Gen2 combines Azure Blob Storage with a hierarchical namespace. LakeXpress leverages ADLS for enterprise-grade data lake storage with optimized analytics performance.

Features:

•Hierarchical namespace for big data analytics
•Fine-grained access control with ACLs
•Optimized for Databricks and Fabric
•High-performance parallel access
•Native integration with Azure analytics services

Destination - MotherDuck

MotherDuck is a serverless analytics platform powered by DuckDB. LakeXpress publishes Parquet files that MotherDuck can query directly with blazing-fast performance.

Publishing method:

External Tables or DuckDB COPY

Features:

•Serverless DuckDB in the cloud
•Direct Parquet file queries
•Hybrid execution (local + cloud)
•SQL analytics without ETL
•Cost-effective pay-per-query

Related Pipelines

Oracle → OneLake → BigQuery

Oracle → S3 → DuckLake

Oracle → OneLake → Databricks

SAP HANA → ABS → MotherDuck

SAP HANA → ADLS → Fabric

Oracle → ADLS → DuckLake