From Teradata to BigQuery using Google Cloud Storage

Configure your LakeXpress data pipeline with the selected components

LakeXpress Orchestrator

Manages end-to-end pipelines
Orchestrates FastBCP extracts with retries & logging
Handles incremental sync and schema-aware metadata

Control DB for increments/custom rules & logsMetadata-drivenLinux or Windows

Data

FastBCPEngine

Parquet

Data or Link

Terminal

./LakeXpress config create \
  -a data/ds_credentials.json \
  --log_db_auth_id log_db_ms \
  --source_db_auth_id datasource_teradata_01 \
  --source_db_name sales \
  --source_schema_name "sales,dim" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --fastbcp_p 2 \
  --n_jobs 4 \
  --target_storage_id gcs_lake_prd \
  --generate_metadata \
  --sub_path /ingest/bronze \
  --incremental_table "sales.orders:o_orderdate:date" \
  --incremental_table "sales.lineitem:l_shipdate:date" \
  --publish_method internal \
  --publish_target bigquery_tgt

# First sync - full load
./LakeXpress sync

# Subsequent syncs - incremental updates (much faster!)
./LakeXpress sync

Documentation

Get LakeXpress

Source - Teradata

Teradata is an enterprise-class data warehousing and analytics platform. LakeXpress uses FastBCP with optimized Teradata connectors for efficient data extraction to Parquet format.

Features:

•Native Teradata .NET driver via FastBCP
•Full support for Teradata-specific types
•Parallel extraction to Parquet files
•Optimized for large-scale data warehouses

Format - Apache Parquet

Parquet is the industry-standard columnar file format for analytics. LakeXpress uses FastBCP to extract data from source databases and convert it to Parquet format, ensuring optimal compression, query performance, and compatibility with all modern data platforms.

Advantages:

•Columnar format optimized for analytics
•Efficient compression (typically 3-10x)
•Schema evolution support
•Predicate pushdown for fast queries
•Universal compatibility with cloud platforms
•Preserves data types and precision

Cloud Stage - Google Cloud Storage

Google Cloud Storage offers unified object storage for developers and enterprises. LakeXpress stages Parquet files in GCS for seamless integration with Google Cloud data platforms.

Features:

•Global edge caching
•Multiple storage classes
•Strong consistency
•Integrated with BigQuery
•Automatic encryption at rest

Destination - BigQuery

Google BigQuery is a serverless, highly scalable data warehouse. LakeXpress loads Parquet files from GCS into BigQuery tables using native bulk loading.

Publishing method:

BigQuery Load Job from GCS

Features:

•Direct load from GCS
•Native Parquet format support
•Automatic schema detection
•Petabyte-scale analytics
•Integration with Google Cloud ecosystem

Related Pipelines

MySQL → GCS → DuckLake

MariaDB → OneLake → BigQuery

Teradata → GCS → Fabric

PostgreSQL → S3 → BigQuery

Teradata → OneLake → DuckLake

Teradata → S3 → MotherDuck