From SQL Server to Fabric using Azure Datalake Gen2
Configure your LakeXpress data pipeline with the selected components
LakeXpress Orchestrator
- Manages end-to-end pipelines
- Orchestrates FastBCP extracts with retries & logging
- Handles incremental sync and schema-aware metadata
./LakeXpress config create \
-a data/ds_credentials.json \
--log_db_auth_id log_db_ms \
--source_db_auth_id datasource_sqlserver_01 \
--source_db_name sales \
--source_schema_name "sales,dim" \
--fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
--fastbcp_p 2 \
--n_jobs 4 \
--target_storage_id adls_lake_prd \
--generate_metadata \
--sub_path /ingest/bronze \
--incremental_table "sales.orders:o_orderdate:date" \
--incremental_table "sales.lineitem:l_shipdate:date" \
--publish_method internal \
--publish_target fabric_tgt
# First sync - full load
./LakeXpress sync
# Subsequent syncs - incremental updates (much faster!)
./LakeXpress syncSource - SQL Server
Microsoft SQL Server is a leading enterprise data platform. LakeXpress leverages FastBCP's advanced techniques to extract SQL Server data with maximum efficiency.
Features:
- •Native SQL Server driver via FastBCP
- •Support for SQL Server-specific data types
- •High-performance parallel extraction
- •Optimized for Windows and Linux environments
Format - Apache Parquet
Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.
Advantages:
- •Columnar format optimized for analytics
- •Efficient compression (typically 3-10x)
- •Schema evolution support
- •Predicate pushdown for fast queries
- •Universal compatibility with cloud platforms
- •Preserves data types and precision
Cloud Stage - Azure Data Lake Storage Gen2
ADLS Gen2 combines Azure Blob Storage with a hierarchical namespace. LakeXpress leverages ADLS for enterprise-grade data lake storage with optimized analytics performance.
Features:
- •Hierarchical namespace for big data analytics
- •Fine-grained access control with ACLs
- •Optimized for Databricks and Fabric
- •High-performance parallel access
- •Native integration with Azure analytics services
Destination - Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution. LakeXpress publishes data directly into Fabric lakehouses or warehouses with native OneLake integration.
Publishing method:
OneLake Delta Lake or SQL Analytics Endpoint
Features:
- •Native OneLake integration
- •Automatic Delta Lake conversion
- •Direct query from Power BI
- •Unified governance
- •Real-time analytics
