From PostgreSQL to Redshift using Google Cloud Storage

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_postgresql_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id gcs_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target redshift_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - PostgreSQL

PostgreSQL is a powerful and robust open-source relational database. LakeXpress optimizes extractions from PostgreSQL using FastBCP's native connector for excellent performance.

Features:

•Direct streaming read from database
•Full support for PostgreSQL data types
•Secure SSL connections
•Parallel extraction to Parquet

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Google Cloud Storage

Google Cloud Storage offers unified object storage for developers and enterprises. LakeXpress stages Parquet files in GCS for seamless integration with Google Cloud data platforms.

Features:

•Global edge caching
•Multiple storage classes
•Strong consistency
•Integrated with BigQuery
•Automatic encryption at rest

Destination - Amazon Redshift

Amazon Redshift is a fast, scalable cloud data warehouse. LakeXpress publishes Parquet files from S3 directly into Redshift using COPY commands for optimal performance.

Publishing method:

COPY FROM S3 with Parquet format

Features:

•Direct COPY FROM S3
•Automatic schema mapping from Parquet
•Massively parallel processing (MPP)
•Columnar storage for fast queries
•Integration with AWS ecosystem

Related Pipelines

PostgreSQL → S3 → Redshift

PostgreSQL → ADLS → BigQuery

SQL Server → GCS → Snowflake

MySQL → GCS → BigQuery

MariaDB → GCS → Fabric

PostgreSQL → GCS → DuckLake