LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

FastBCPEngine

lakexpress— bash

1$./LakeXpress config create \
2  --auth_file data/ds_credentials.json \
3  --log_db_auth_id log_db_ms \
4  --source_db_auth_id datasource_postgresql_01 \
5  --source_db_name sales \
6  --source_schema_name "sales,dim" \
7  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
8  --fastbcp_p 2 \
9  --n_jobs 4 \
10  --target_storage_id onelake_lake_prd \
11  --generate_metadata \
12  --sub_path /ingest/bronze \
13  --incremental_table "sales.orders:o_orderdate:date" \
14  --incremental_table "sales.lineitem:l_shipdate:date" \
15  --publish_method internal \
16  --publish_target databricks_tgt \
17  --sync_id move_sales_to_Databricks-prod
18
19# First sync - full load
20$./LakeXpress sync \
21  --auth_file data/ds_credentials.json \
22  --log_db_auth_id log_db_ms \
23  --sync_id move_sales_to_Databricks-prod
24
25# Subsequent syncs - incremental updates (much faster!)
26$./LakeXpress sync \
27  --auth_file data/ds_credentials.json \
28  --log_db_auth_id log_db_ms \
29  --sync_id move_sales_to_Databricks-prod

Documentation

Get LakeXpress

Source - PostgreSQL

PostgreSQL is a powerful and robust open-source relational database. LakeXpress optimizes extractions from PostgreSQL using FastBCP's native connector for excellent performance.

Features:

Direct streaming read from database
Full support for PostgreSQL data types
Secure SSL connections
Parallel extraction to Parquet

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

Columnar format optimized for analytics
Efficient compression (typically 3-10x)
Schema evolution support
Predicate pushdown for fast queries
Universal compatibility with cloud platforms
Preserves data types and precision

Cloud Stage - Microsoft OneLake

OneLake is Microsoft Fabric's unified data lake. LakeXpress stages data directly in OneLake, providing native integration with all Fabric workloads and eliminating data duplication.

Features:

Native Fabric integration
Single copy of data across all workloads
Automatic Delta Lake format
Built-in governance and security
Seamless data sharing across Fabric

Destination - Databricks

Databricks provides a unified analytics platform built on Apache Spark. LakeXpress publishes Parquet files as Delta Lake tables for optimal lakehouse performance.