From SQL Server to Databricks using Azure Blob Storage

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_sqlserver_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id abs_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target databricks_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - SQL Server

Microsoft SQL Server is a leading enterprise data platform. LakeXpress leverages FastBCP's advanced techniques to extract SQL Server data with maximum efficiency.

Features:

•Native SQL Server driver via FastBCP
•Support for SQL Server-specific data types
•High-performance parallel extraction
•Optimized for Windows and Linux environments

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Azure Blob Storage

Azure Blob Storage is Microsoft's object storage solution for the cloud. LakeXpress uses Azure Blob as an intermediate staging layer for Parquet files with high performance and reliability.

Features:

•Massive scalability
•Multiple access tiers (Hot, Cool, Archive)
•Integrated with Azure ecosystem
•Built-in encryption and security
•Lifecycle management policies

Destination - Databricks

Databricks provides a unified analytics platform built on Apache Spark. LakeXpress publishes Parquet files as Delta Lake tables for optimal lakehouse performance.

Publishing method:

Delta Lake MERGE or COPY INTO

Features:

•Native Parquet and Delta Lake support
•ACID transactions
•Time travel and versioning
•Optimized for analytical queries
•Unity Catalog integration

Related Pipelines

SQL Server → OneLake → Databricks

SQL Server → S3 → MotherDuck

SQL Server → ABS → BigQuery

SQL Server → ADLS → BigQuery

SQL Server → GCS → Snowflake

MariaDB → S3 → Databricks