LakeXpress

What is LakeXpress?

LakeXpress is a CLI tool that exports database tables to partitioned Parquet files and publishes them to cloud data platforms. It uses FastBCP to stream data in parallel without exhausting memory.

Architecture

LakeXpress connects four components in a data pipeline:

Source Database

The database you want to export data from. LakeXpress reads tables (not views) from the source and converts them to Parquet files.

Supported sources

Connection setup and required privileges for each source database

• Oracle
• PostgreSQL
• SQL Server
• MySQL
• MariaDB
• SAP HANA
• Teradata

Intermediate Storage

Parquet files are written to a storage backend, where they can be used as they are or published to a target platform. This can be local disk or cloud object storage.

Supported storage backends

Credentials and paths for each storage backend. S3-compatible storage (MinIO, OVH...) uses the AWS S3 page

• Local filesystem
• AWS S3
• Google Cloud Storage
• Azure Blob Storage
• Microsoft Fabric OneLake

Target Platform

Optional. Once Parquet files are in storage, LakeXpress can publish them to a cloud data platform or catalog, either as external tables (the data stays in storage) or as internal tables (the data is loaded into the platform). See Compatibility for the methods each platform supports.

Supported platforms

Setup, authentication and options for each platform

• Snowflake
• Databricks
• Microsoft Fabric
• Amazon Redshift
• BigQuery
• MotherDuck
• AWS Glue
• DuckLake

LakeXpress DB

A dedicated database where LakeXpress stores its sync configurations, the history and status of every run, and the watermarks of incremental syncs. Every LakeXpress command uses it, so it must be set up before the first export. It also enables tracking runs and resuming failed exports.

See Set up the LakeXpress DB for an overview.

Supported LakeXpress DBs

How to create and initialize the LakeXpress DB on each database

• PostgreSQL
• SQL Server
• MySQL
• SQLite
• DuckDB

What you'll need

1. LakeXpress binary

   Download and unzip the package for Windows or Linux, then save the license file you received by email.

   • Installation guide
2. Source database user

   A read-only user with SELECT on the schemas to export.

   • Database setup

   PostgreSQLSQL ServerOracleMySQLMariaDBSAP HANATeradata
3. LakeXpress DB

   An empty PostgreSQL, SQL Server or MySQL database owned by its user, or a SQLite or DuckDB file (nothing to create).

   • Set up the LakeXpress DB

   PostgreSQLSQL ServerMySQLSQLiteDuckDB
4. Storage destination

   A local directory, or credentials for cloud object storage.

   • Storage config

   Local filesystemAmazon S3Google Cloud StorageAzure Blob StorageOneLake
5. Publishing targetOptional

   Credentials for the platform that creates tables over the exported Parquet files. Skip it to keep the files in storage only.

   • Publishing guides

   SnowflakeDatabricksMicrosoft FabricAmazon RedshiftBigQueryMotherDuckAWS GlueDuckLake

Key features

• Cross-platformNative binaries for Windows and Linux
• Parallel exportsMultiple tables at once, with per-table partitioning
• Incremental syncWatermark-based delta exports
• Schema filteringInclude/exclude schemas and tables via SQL patterns
• Resume on failurePick up where a failed export left off
• CDM metadataGenerate Common Data Model files

How it works

1. 1InitializeLakeXpress lxdb init creates the LakeXpress tables in the LakeXpress DB (once)
2. 2ConfigureLakeXpress config create defines your source database, storage target, and optional publishing
3. 3SyncLakeXpress sync exports tables to Parquet and, if a publishing target is configured, publishes them

See the Quick Start Guide for a full walkthrough with real commands.