From SAP HANA to AWS Glue using Microsoft OneLake

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_sap-hana_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id onelake_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target aws-glue_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - SAP HANA

SAP HANA is a high-performance in-memory database platform. LakeXpress optimizes extractions from HANA using FastBCP to leverage its in-memory capabilities.

Features:

•Native SAP HANA connector via FastBCP
•Support for views and columnar tables
•Optimized for in-memory data
•High-speed parallel extraction

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Microsoft OneLake

OneLake is Microsoft Fabric's unified data lake. LakeXpress stages data directly in OneLake, providing native integration with all Fabric workloads and eliminating data duplication.

Features:

•Native Fabric integration
•Single copy of data across all workloads
•Automatic Delta Lake format
•Built-in governance and security
•Seamless data sharing across Fabric

Destination - AWS Glue

AWS Glue is a serverless data integration service. LakeXpress publishes Parquet files to S3 and registers tables in the Glue Data Catalog for seamless query access.

Publishing method:

S3 + Glue Catalog Table Registration

Features:

•Automatic schema registry in Glue Catalog
•Integration with Athena for SQL queries
•Crawlers for metadata discovery
•Partition management
•EMR and Redshift Spectrum compatibility

Related Pipelines

Oracle → OneLake → Redshift

MySQL → OneLake → Fabric

SAP HANA → S3 → Snowflake

PostgreSQL → OneLake → Snowflake

SAP HANA → ADLS → Databricks

SAP HANA → S3 → DuckLake