LakeXpress

What is LakeXpress?

LakeXpress is a CLI tool that exports database tables to partitioned Parquet files and publishes them to cloud data platforms. It uses FastBCP to stream data in parallel without exhausting memory.

Architecture

LakeXpress connects four components in a data pipeline:

Source Database

The database you want to export data from. LakeXpress reads tables (not views) from the source and converts them to Parquet files.

Supported sources

Connection setup and required privileges for each source database

• Oracle
• PostgreSQL
• SQL Server
• MySQL
• MariaDB
• SAP HANA
• Teradata

Intermediate Storage

Parquet files are written to a storage backend, where they can be used as they are or published to a target platform. This can be local disk or cloud object storage.

Supported storage backends

Credentials and paths for each storage backend. S3-compatible storage (MinIO, OVH...) uses the AWS S3 page

• Local filesystem
• AWS S3
• Google Cloud Storage
• Azure Blob Storage
• Microsoft Fabric OneLake

Target Platform

Optional. Once Parquet files are in storage, LakeXpress can publish them to a cloud data platform or catalog, either as external tables (the data stays in storage) or as internal tables (the data is loaded into the platform). See Compatibility for the methods each platform supports.

Supported platforms

Setup, authentication and options for each platform

• Snowflake
• Databricks
• Microsoft Fabric
• Amazon Redshift
• BigQuery
• MotherDuck
• AWS Glue
• DuckLake

LakeXpress DB

A dedicated database where LakeXpress stores its sync configurations, the history and status of every run, and the watermarks of incremental syncs. Every LakeXpress command uses it, so it must be set up before the first export. It also enables tracking runs and resuming failed exports.

See Set up the LakeXpress DB for an overview.

Supported LakeXpress DBs

How to create and initialize the LakeXpress DB on each database

• PostgreSQL
• SQL Server
• MySQL
• SQLite
• DuckDB

What you'll need

Component	What to do	Details
LakeXpress binary	Download and unzip	Installation guide
Source database user	Create a read-only user with SELECT on target schemas	Database setup
LakeXpress DB	An empty PostgreSQL, SQL Server or MySQL database owned by its user, or a SQLite or DuckDB file (nothing to create)	Set up the LakeXpress DB
Storage destination	A local directory or cloud storage credentials	Storage config
Publishing target (optional)	Credentials for Snowflake, Databricks, etc.	Snowflake · Databricks · more...

Key features

• Cross-platformNative binaries for Windows and Linux
• Parallel exportsMultiple tables at once, with per-table partitioning
• Incremental syncWatermark-based delta exports
• Schema filteringInclude/exclude schemas and tables via SQL patterns
• Resume on failurePick up where a failed export left off
• CDM metadataGenerate Common Data Model files

How it works

1. 1InitializeLakeXpress lxdb init creates the LakeXpress tables in the LakeXpress DB (once)
2. 2ConfigureLakeXpress config create defines your source database, storage target, and optional publishing
3. 3SyncLakeXpress sync exports tables to Parquet and, if a publishing target is configured, publishes them

See the Quick Start Guide for a full walkthrough with real commands.