From SQL Server to MotherDuck using AWS S3

Configure your LakeXpress data pipeline with the selected components

LX

LakeXpress Orchestrator

  • Manages end-to-end pipelines
  • Orchestrates FastBCP extracts with retries & logging
  • Handles incremental sync and schema-aware metadata
Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows
FastBCP
FastBCPEngine
Terminal
./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_sqlserver_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id s3_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target motherduck_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync
Get LakeXpress

Source - SQL Server

Microsoft SQL Server is a leading enterprise data platform. LakeXpress leverages FastBCP's advanced techniques to extract SQL Server data with maximum efficiency.

Features:

  • Native SQL Server driver via FastBCP
  • Support for SQL Server-specific data types
  • High-performance parallel extraction
  • Optimized for Windows and Linux environments

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

  • Columnar format optimized for analytics
  • Efficient compression (typically 3-10x)
  • Schema evolution support
  • Predicate pushdown for fast queries
  • Universal compatibility with cloud platforms
  • Preserves data types and precision

Cloud Stage - Amazon Simple Storage Service

AWS S3 is the industry-leading object storage service. LakeXpress stages Parquet files in S3 for scalable, durable, and cost-effective cloud storage before publishing to your target platform.

Features:

  • Unlimited scalability
  • 99.999999999% durability
  • Lifecycle policies for cost optimization
  • Server-side encryption
  • Integration with AWS Glue Data Catalog

Destination - MotherDuck

MotherDuck is a serverless analytics platform powered by DuckDB. LakeXpress publishes Parquet files that MotherDuck can query directly with blazing-fast performance.

Publishing method:

External Tables or DuckDB COPY

Features:

  • Serverless DuckDB in the cloud
  • Direct Parquet file queries
  • Hybrid execution (local + cloud)
  • SQL analytics without ETL
  • Cost-effective pay-per-query