From MariaDB to Snowflake using AWS S3

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_mariadb_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id s3_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target snowflake_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - MariaDB

MariaDB is a MySQL fork offering advanced features and better performance. LakeXpress ensures full compatibility with MariaDB through FastBCP.

Features:

•Full MySQL compatibility
•Support for advanced MariaDB features
•Optimized extraction performance
•Parallel processing to Parquet

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Amazon Simple Storage Service

AWS S3 is the industry-leading object storage service. LakeXpress stages Parquet files in S3 for scalable, durable, and cost-effective cloud storage before publishing to your target platform.

Features:

•Unlimited scalability
•99.999999999% durability
•Lifecycle policies for cost optimization
•Server-side encryption
•Integration with AWS Glue Data Catalog

Destination - Snowflake

Snowflake is a cloud data platform built for analytics. LakeXpress publishes Parquet files from cloud storage directly into Snowflake using COPY INTO commands for optimal performance.

Publishing method:

COPY INTO with MATCH_BY_COLUMN_NAME

Features:

•Direct COPY FROM stage (S3, Azure, GCS)
•Automatic schema detection from Parquet
•Zero-copy cloning capabilities
•Automatic clustering and optimization
•Support for semi-structured data

Related Pipelines

SQL Server → OneLake → Snowflake

MariaDB → ABS → MotherDuck

MariaDB → ADLS → Redshift

Oracle → S3 → Redshift

Teradata → S3 → Databricks

MySQL → S3 → AWS Glue