# LakeXpress Documentation

> An efficient database-to-Parquet export tool with multi-cloud object storage support. Export from PostgreSQL, SQL Server, MySQL, Oracle to S3, Azure, GCS, OneLake, Snowflake, Databricks, and more.


## 0.3

Fast database-to-Parquet export tool with multi-cloud object storage support.

- [LakeXpress](/0.3.md): Fast database-to-Parquet export tool with multi-cloud object storage support.

### advanced

- [Export Performance](/0.3/advanced/export-performance.md): Configure export methods and parallel processing for high-performance data export.
- [Incremental Sync](/0.3/advanced/incremental-sync.md): Sync only changed data using watermark-based incremental exports.
- [Examples](/0.3/advanced/incremental/examples.md): Complete step-by-step examples and real-world scenarios for incremental sync.
- [Loading Strategies](/0.3/advanced/incremental/loading-strategies.md): Choose between append and upsert strategies for incremental data loading.
- [Troubleshooting](/0.3/advanced/incremental/troubleshooting.md): Troubleshoot common incremental sync issues and optimize performance.
- [Watermarks](/0.3/advanced/incremental/watermarks.md): Understand watermark tracking, safety lag, querying, and resetting watermarks.

### compatibility

Supported databases, storage backends, and publishing targets for LakeXpress.

- [Compatibility](/0.3/compatibility.md): Supported databases, storage backends, and publishing targets for LakeXpress.

### configuration

- [Database Connections](/0.3/configuration/database-configuration.md): Configure database connections for LakeXpress source databases and the LakeXpress DB.
- [DuckDB (LakeXpress DB Only)](/0.3/configuration/databases/duckdb.md): Configure DuckDB as a LakeXpress DB.
- [MariaDB](/0.3/configuration/databases/mariadb.md): Configure MariaDB connections for LakeXpress source databases.
- [MySQL](/0.3/configuration/databases/mysql.md): Configure MySQL connections for LakeXpress source databases and the LakeXpress DB.
- [Oracle](/0.3/configuration/databases/oracle.md): Configure Oracle connections for LakeXpress source databases.
- [PostgreSQL](/0.3/configuration/databases/postgresql.md): Configure PostgreSQL connections for LakeXpress source databases and the LakeXpress DB.
- [SAP HANA](/0.3/configuration/databases/sap-hana.md): Configure SAP HANA connections for LakeXpress source databases.
- [SQL Server](/0.3/configuration/databases/sql-server.md): Configure SQL Server connections for LakeXpress source databases and the LakeXpress DB.
- [SQLite (LakeXpress DB Only)](/0.3/configuration/databases/sqlite.md): Configure SQLite as a LakeXpress DB.
- [Teradata](/0.3/configuration/databases/teradata.md): Configure Teradata connections for LakeXpress source databases.
- [Azure Storage](/0.3/configuration/storage/azure.md): Configure Azure Data Lake Storage Gen2 and Blob Storage for LakeXpress exports.
- [Google Cloud Storage (GCS)](/0.3/configuration/storage/gcs.md): Configure Google Cloud Storage for LakeXpress Parquet file exports.
- [Local Filesystem](/0.3/configuration/storage/local-filesystem.md): Export Parquet files to local directories with LakeXpress.
- [Microsoft Fabric OneLake](/0.3/configuration/storage/onelake.md): Configure Microsoft Fabric OneLake storage for LakeXpress Parquet file exports.
- [S3-Compatible Storage](/0.3/configuration/storage/s3.md): Configure AWS S3 and S3-compatible storage (MinIO, OVH, etc.) for LakeXpress exports.
- [Intermediate Storage](/0.3/configuration/storage/storage-overview.md): Configure local and cloud storage backends for LakeXpress Parquet file exports.

### guides

- [Examples & Recipes](/0.3/guides/command-cookbook.md): Tested commands and real-world examples for common LakeXpress scenarios covering exports, publishing, and incremental sync.
- [Quick start guide](/0.3/guides/quickstart.md): Get started with LakeXpress in two steps -- create a config and run a sync.

### installation

Install LakeXpress on Windows, Linux, or macOS. Download, configure, and start building database-to-lakehouse pipelines quickly.

- [Installation](/0.3/installation.md): Install LakeXpress on Windows, Linux, or macOS. Download, configure, and start building database-to-lakehouse pipelines quickly.

### integrations

- [BigQuery Publishing](/0.3/integrations/bigquery-publishing.md): Create BigQuery external and native tables from Parquet files stored in Google Cloud Storage.
- [BigQuery Reference](/0.3/integrations/bigquery-reference.md): Data type mappings, CLI arguments, querying, and troubleshooting for BigQuery publishing.
- [Databricks Publishing](/0.3/integrations/databricks-publishing.md): Create Databricks Unity Catalog external and managed Delta tables from exported Parquet files.
- [Databricks Reference](/0.3/integrations/databricks-reference.md): Data type mappings, CLI arguments, querying tables, and Delta features for Databricks publishing.
- [ducklake-publishing](/0.3/integrations/ducklake-publishing.md): Publish database exports as DuckLake tables with LakeXpress. Register Parquet files in a DuckLake catalog for SQL queries.
- [Microsoft Fabric Publishing](/0.3/integrations/fabric-publishing.md): Create Microsoft Fabric Lakehouse Delta tables and SQL analytics views from exported Parquet files.
- [Microsoft Fabric Reference](/0.3/integrations/fabric-reference.md): Data type mappings, querying tables, troubleshooting, and CLI arguments for Fabric publishing.
- [AWS Glue Publishing](/0.3/integrations/glue-publishing.md): Create AWS Glue Data Catalog tables from exported Parquet files for use with Athena, Redshift Spectrum, and EMR.
- [AWS Glue Reference](/0.3/integrations/glue-reference.md): Data type mappings, CLI arguments, and querying for AWS Glue publishing.
- [MotherDuck Publishing](/0.3/integrations/motherduck-publishing.md): Create MotherDuck tables and views from exported Parquet files in S3, GCS, or Azure storage.
- [MotherDuck Reference](/0.3/integrations/motherduck-reference.md): Data type mappings, CLI arguments, querying, and troubleshooting for MotherDuck publishing.
- [Amazon Redshift Publishing](/0.3/integrations/redshift-publishing.md): Create Amazon Redshift internal and external tables from exported Parquet files using LakeXpress.
- [Snowflake Publishing](/0.3/integrations/snowflake-publishing.md): Create Snowflake external and internal tables from exported Parquet files using LakeXpress.
- [Snowflake Reference](/0.3/integrations/snowflake-reference.md): Data type mappings, CLI arguments, and deprecated options for Snowflake publishing.

### reference

- [CLI Reference](/0.3/reference/cli-reference.md): Complete reference for all LakeXpress commands, options, and configuration parameters.
- [Logging and Monitoring](/0.3/reference/logging.md): Database and file-based logging systems for tracking LakeXpress export activity.

### sitemap

Complete overview of all LakeXpress documentation pages.

- [Sitemap](/0.3/sitemap.md): Complete overview of all LakeXpress documentation pages.

### wizard

Build your LakeXpress command interactively.

- [Command Wizard](/0.3/wizard.md): Build your LakeXpress command interactively.

## latest

Fast database-to-Parquet export tool with multi-cloud object storage support.

- [LakeXpress](/latest.md): Fast database-to-Parquet export tool with multi-cloud object storage support.

### advanced

- [Export Performance](/latest/advanced/export-performance.md): Configure export methods and parallel processing for high-performance data export.
- [Incremental Sync](/latest/advanced/incremental-sync.md): Sync only changed data using watermark-based incremental exports.
- [Loading Strategies](/latest/advanced/incremental/loading-strategies.md): Choose between the append, upsert and delete_insert strategies for incremental data loading.
- [Troubleshooting](/latest/advanced/incremental/troubleshooting.md): Troubleshoot common incremental sync issues and optimize performance.
- [Watermarks](/latest/advanced/incremental/watermarks.md): Understand watermark tracking, late-arriving data, safety lag, querying, and resetting watermarks.

### ai

- [LakeXpress Agent Skill](/latest/ai/agent-skills.md): Install the official arpe.io agent skill that teaches Claude Code, Codex, Cursor, Copilot and other AI agents to build correct LakeXpress command lines
- [LakeXpress MCP Server](/latest/ai/mcp-server.md): Connect Claude, ChatGPT, Cursor, VS Code and other AI assistants to the arpe.io MCP server to build, check and run LakeXpress commands

### cli

LakeXpress commands, the options they share, and license resolution.

- [CLI Reference](/latest/cli.md): LakeXpress commands, the options they share, and license resolution.
- [cleanup](/latest/cli/cleanup.md): Remove stale or failed runs of a LakeXpress sync configuration.
- [config](/latest/cli/config.md): Create, list and delete LakeXpress sync configurations.
- [lxdb](/latest/cli/lxdb.md): Initialize, empty, drop and unlock the LakeXpress DB.
- [run](/latest/cli/run.md): Run a LakeXpress export from a legacy YAML configuration file.
- [status](/latest/cli/status.md): Show the runs of a LakeXpress sync configuration and their status.
- [sync](/latest/cli/sync.md): Run a LakeXpress sync configuration, or only its export or publish step.

### configuration

- [Database Connections](/latest/configuration/database-configuration.md): Configure database connections for LakeXpress source databases and the LakeXpress DB.
- [MariaDB](/latest/configuration/databases/mariadb.md): Configure MariaDB connections for LakeXpress source databases.
- [MySQL](/latest/configuration/databases/mysql.md): Configure a MySQL source database for LakeXpress.
- [Oracle](/latest/configuration/databases/oracle.md): Configure Oracle connections for LakeXpress source databases.
- [PostgreSQL](/latest/configuration/databases/postgresql.md): Configure a PostgreSQL source database for LakeXpress.
- [SAP HANA](/latest/configuration/databases/sap-hana.md): Configure SAP HANA connections for LakeXpress source databases.
- [SQL Server](/latest/configuration/databases/sql-server.md): Configure a SQL Server source database for LakeXpress.
- [Teradata](/latest/configuration/databases/teradata.md): Configure Teradata connections for LakeXpress source databases.
- [Azure Storage](/latest/configuration/storage/azure.md): Configure Azure Data Lake Storage Gen2 and Blob Storage for LakeXpress exports.
- [Google Cloud Storage (GCS)](/latest/configuration/storage/gcs.md): Configure Google Cloud Storage for LakeXpress Parquet file exports.
- [Local Filesystem](/latest/configuration/storage/local-filesystem.md): Export Parquet files to local directories with LakeXpress.
- [Microsoft Fabric OneLake](/latest/configuration/storage/onelake.md): Configure Microsoft Fabric OneLake storage for LakeXpress Parquet file exports.
- [S3-Compatible Storage](/latest/configuration/storage/s3.md): Configure AWS S3 and S3-compatible storage (MinIO, OVH, etc.) for LakeXpress exports.
- [Intermediate Storage](/latest/configuration/storage/storage-overview.md): Configure local and cloud storage backends for LakeXpress Parquet file exports.

### guides

- [Examples & Recipes](/latest/guides/command-cookbook.md): Tested commands and real-world examples for common LakeXpress scenarios covering exports, publishing, and incremental sync.
- [Quick start guide](/latest/guides/quickstart.md): Your first LakeXpress export -- from a PostgreSQL database to Parquet files on local disk.
- [Azure Storage Exports](/latest/guides/recipes/azure.md): Export tables to Parquet files on Azure Blob Storage and ADLS Gen2.
- [BigQuery Publishing](/latest/guides/recipes/bigquery.md): Export to cloud storage and publish the tables to BigQuery.
- [Configuration Management](/latest/guides/recipes/configuration-management.md): Create, list and delete sync configurations, check their status, and run legacy YAML files.
- [Databricks Publishing](/latest/guides/recipes/databricks.md): Export to cloud storage and publish the tables to Databricks Unity Catalog.
- [DuckLake Publishing](/latest/guides/recipes/ducklake.md): Export to cloud storage and publish the tables to DuckLake.
- [Microsoft Fabric Publishing](/latest/guides/recipes/fabric.md): Export to OneLake and publish the tables to Microsoft Fabric.
- [Google Cloud Storage Exports](/latest/guides/recipes/gcs.md): Export tables to Parquet files on Google Cloud Storage.
- [AWS Glue Publishing](/latest/guides/recipes/glue.md): Export to S3 and register the tables in the AWS Glue Data Catalog.
- [Incremental Sync](/latest/guides/recipes/incremental-sync.md): Export only new or changed rows on each sync, with watermark columns.
- [LakeXpress DB Commands](/latest/guides/recipes/lakexpress-db.md): Initialize, empty, drop and unlock the LakeXpress DB, for each supported backend.
- [Local Exports](/latest/guides/recipes/local-exports.md): Export PostgreSQL, SQL Server, MySQL, MariaDB and Teradata tables to Parquet files on local disk.
- [MotherDuck Publishing](/latest/guides/recipes/motherduck.md): Export to cloud storage and publish the tables to MotherDuck.
- [OneLake Exports](/latest/guides/recipes/onelake.md): Export tables to Parquet files on Microsoft OneLake.
- [Amazon Redshift Publishing](/latest/guides/recipes/redshift.md): Export to cloud storage and publish the tables to Amazon Redshift.
- [Resume Failed Exports](/latest/guides/recipes/resume-failed-exports.md): Retry only the tables that failed in a previous run.
- [AWS S3 Exports](/latest/guides/recipes/s3.md): Export tables to Parquet files on AWS S3 and S3-compatible storage.
- [Snowflake Publishing](/latest/guides/recipes/snowflake.md): Export to cloud storage and publish the tables to Snowflake, as external or internal tables.
- [Table Filtering](/latest/guides/recipes/table-filtering.md): Select which tables to export with include and exclude patterns.

### installation

Install LakeXpress on Windows or Linux. Download, configure, and start building database-to-lakehouse pipelines quickly.

- [Installation](/latest/installation.md): Install LakeXpress on Windows or Linux. Download, configure, and start building database-to-lakehouse pipelines quickly.

### integrations

- [BigQuery Publishing](/latest/integrations/bigquery-publishing.md): Create BigQuery external and native tables from Parquet files stored in Google Cloud Storage.
- [BigQuery Reference](/latest/integrations/bigquery-reference.md): Data type mappings, CLI arguments, querying, and troubleshooting for BigQuery publishing.
- [Databricks Publishing](/latest/integrations/databricks-publishing.md): Create Databricks Unity Catalog external and managed Delta tables from exported Parquet files.
- [Databricks Reference](/latest/integrations/databricks-reference.md): Data type mappings, CLI arguments, querying tables, and Delta features for Databricks publishing.
- [DuckLake Publishing](/latest/integrations/ducklake-publishing.md): Publish database exports as DuckLake tables with LakeXpress. Register Parquet files in a DuckLake catalog for SQL queries.
- [Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md): Create Microsoft Fabric Lakehouse Delta tables and SQL analytics views from exported Parquet files.
- [Microsoft Fabric Reference](/latest/integrations/fabric-reference.md): Data type mappings, querying tables, troubleshooting, and CLI arguments for Fabric publishing.
- [AWS Glue Publishing](/latest/integrations/glue-publishing.md): Create AWS Glue Data Catalog tables from exported Parquet files for use with Athena, Redshift Spectrum, and EMR.
- [AWS Glue Reference](/latest/integrations/glue-reference.md): Data type mappings, CLI arguments, and querying for AWS Glue publishing.
- [MotherDuck Publishing](/latest/integrations/motherduck-publishing.md): Create MotherDuck tables and views from exported Parquet files in S3, GCS, or Azure storage.
- [MotherDuck Reference](/latest/integrations/motherduck-reference.md): Data type mappings, CLI arguments, querying, and troubleshooting for MotherDuck publishing.
- [Amazon Redshift Publishing](/latest/integrations/redshift-publishing.md): Create Amazon Redshift internal and external tables from exported Parquet files using LakeXpress.
- [Amazon Redshift Reference](/latest/integrations/redshift-reference.md): Data type mappings, CLI arguments, and querying for Amazon Redshift publishing.
- [Snowflake Publishing](/latest/integrations/snowflake-publishing.md): Create Snowflake external and internal tables from exported Parquet files using LakeXpress.
- [Snowflake Reference](/latest/integrations/snowflake-reference.md): Data type mappings, CLI arguments, and deprecated options for Snowflake publishing.

### lakexpress-db

Create the LakeXpress DB, the database where LakeXpress stores its sync configurations and run history.

- [Set up the LakeXpress DB](/latest/lakexpress-db.md): Create the LakeXpress DB, the database where LakeXpress stores its sync configurations and run history.
- [DuckDB as LakeXpress DB](/latest/lakexpress-db/duckdb.md): Use a DuckDB file as the LakeXpress DB.
- [MySQL as LakeXpress DB](/latest/lakexpress-db/mysql.md): Create a MySQL database to use as the LakeXpress DB.
- [PostgreSQL as LakeXpress DB](/latest/lakexpress-db/postgresql.md): Create a PostgreSQL database to use as the LakeXpress DB.
- [SQL Server as LakeXpress DB](/latest/lakexpress-db/sql-server.md): Create a SQL Server database to use as the LakeXpress DB.
- [SQLite as LakeXpress DB](/latest/lakexpress-db/sqlite.md): Use a SQLite file as the LakeXpress DB.

### reference

- [CLI Commands](/latest/reference/cli-commands.md): Every LakeXpress command on one page, with a link to its reference.
- [Compatibility](/latest/reference/compatibility.md): Supported databases, storage backends, and publishing targets for LakeXpress.
- [Exit Codes](/latest/reference/exit-codes.md): Process exit codes returned by LakeXpress, for schedulers and scripts.
- [Logging and Monitoring](/latest/reference/logging.md): Database and file-based logging systems for tracking LakeXpress export activity.

### sitemap

Complete overview of all LakeXpress documentation pages.

- [Sitemap](/latest/sitemap.md): Complete overview of all LakeXpress documentation pages.

### wizard

Build your LakeXpress command interactively.

- [Command Wizard](/latest/wizard.md): Build your LakeXpress command interactively.


---

# Full Documentation Content

# LakeXpress

## What is LakeXpress?[​](#what-is-lakexpress "Direct link to What is LakeXpress?")

LakeXpress is a CLI tool that exports database tables to partitioned Parquet files and publishes them to cloud data platforms. It uses [FastBCP](https://www.arpe.io/fastbcp/?v=82a9e4d26595) to stream data in parallel without exhausting memory.

## Architecture[​](#architecture "Direct link to Architecture")

LakeXpress connects four components in a data pipeline:

### Source Database[​](#source-database "Direct link to Source Database")

The database you want to export data from. LakeXpress reads tables and views from the source and converts them to Parquet files.

Supported sources: Oracle, PostgreSQL, SQL Server, MySQL, MariaDB, SAP HANA, Teradata.

### Intermediate Storage[​](#intermediate-storage "Direct link to Intermediate Storage")

Parquet files are written to a storage backend before being registered in a target platform. This can be local disk or cloud object storage.

Supported backends: local filesystem, AWS S3, S3-compatible (MinIO, etc.), Google Cloud Storage, Azure Blob Storage.

### Target Platform[​](#target-platform "Direct link to Target Platform")

Once Parquet files are in storage, LakeXpress registers them as external tables in a cloud data platform or catalog.

Supported platforms: Snowflake, Databricks, Microsoft Fabric, Amazon Redshift, BigQuery, MotherDuck, AWS Glue, DuckLake.

### LakeXpress DB[​](#lakexpress-db "Direct link to LakeXpress DB")

A dedicated database where LakeXpress logs run history, job metadata, and exported file details. This enables tracking, auditing, and resuming failed exports.

Supported databases: PostgreSQL, SQL Server, MySQL, SQLite, DuckDB.

## What you'll need[​](#what-youll-need "Direct link to What you'll need")

| Component                          | What to do                                                      | Details                                                                                                                                                                 |
| ---------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **LakeXpress binary**              | Download and unzip                                              | [Installation guide](/0.3/installation.md)                                                                                                                              |
| **Source database user**           | Create a read-only user with SELECT on target schemas           | [Database setup](/0.3/configuration/database-configuration.md)                                                                                                          |
| **LakeXpress DB**                  | A new or existing database with a user that has full privileges | [Database setup](/0.3/configuration/database-configuration.md#lakexpress-db)                                                                                            |
| **Storage destination**            | A local directory or cloud storage credentials                  | [Storage config](/0.3/configuration/storage/storage-overview.md)                                                                                                        |
| **Publishing target** *(optional)* | Credentials for Snowflake, Databricks, etc.                     | [Snowflake](/0.3/integrations/snowflake-publishing.md) · [Databricks](/0.3/integrations/databricks-publishing.md) · [more...](/0.3/integrations/bigquery-publishing.md) |

## Key features[​](#key-features "Direct link to Key features")

* Cross-platformNative binaries for Windows and Linux
* Parallel exportsMultiple tables at once, with per-table partitioning
* Incremental syncWatermark-based delta exports
* Schema filteringInclude/exclude schemas and tables via SQL patterns
* Resume on failurePick up where a failed export left off
* CDM metadataGenerate Common Data Model files

## How it works[​](#how-it-works "Direct link to How it works")

1. 1Configure`LakeXpress config create` defines your source database, storage target, and optional publishing
2. 2Sync`LakeXpress sync` exports tables to Parquet and publishes to your catalog

See the [Quick Start Guide](/0.3/guides/quickstart.md) for a full walkthrough with real commands.

## Next steps[​](#next-steps "Direct link to Next steps")

Ready to export?

[Get Started](./guides/quickstart)[CLI Reference](./reference/cli-reference)[Command Cookbook](./guides/command-cookbook)


---

# Export Performance

LakeXpress auto-detects the best export method per table. Override when needed using CLI flags or the `partition_columns` table in the LakeXpress DB.

## Export Methods[​](#export-methods "Direct link to Export Methods")

### Without Key Column[​](#without-key-column "Direct link to Without Key Column")

| Method      | Database   | Description                             |
| ----------- | ---------- | --------------------------------------- |
| **None**    | All        | Sequential export (no parallelism)      |
| **Ctid**    | PostgreSQL | Tuple ID-based parallel export          |
| **Rowid**   | Oracle     | Row ID-based parallel export            |
| **Physloc** | SQL Server | Physical location-based parallel export |

### With Key Column[​](#with-key-column "Direct link to With Key Column")

| Method            | Description                                          | Key Column                                       |
| ----------------- | ---------------------------------------------------- | ------------------------------------------------ |
| **Random**        | Distributes using column values                      | Integer or bigint                                |
| **DataDriven**    | Splits on distinct values                            | Column with distinct values (e.g., year, region) |
| **RangeId**       | Range partitioning via min/max                       | Numeric or date column                           |
| **Ntile**         | Even distribution                                    | Numeric column                                   |
| **Timepartition** | Partitions by date/time column into Hive-style paths | Date/datetime column: `(col,year,month)`         |

### Timepartition[​](#timepartition "Direct link to Timepartition")

Splits data into time-based chunks where each parallel thread handles a specific time period. Works with any source database -- ideal for time-series data and large historical tables.

**Key column format** -- a parenthesized list of a date column followed by granularity levels:

| Format                    | Output paths                 |
| ------------------------- | ---------------------------- |
| `(col, year)`             | `year=YYYY/`                 |
| `(col, year, month)`      | `year=YYYY/month=MM/`        |
| `(col, year, month, day)` | `year=YYYY/month=MM/day=DD/` |
| `(col, year, week)`       | `year=YYYY/week=WW/`         |

Exported files follow Hive-style partitioned paths, e.g. `year=2008/month=01/CARRIER_CLAIMS.parquet`.

## Tuning Parameters[​](#tuning-parameters "Direct link to Tuning Parameters")

| Parameter                 | Default | Description                                      |
| ------------------------- | ------- | ------------------------------------------------ |
| `--n_jobs`                | 4       | Tables exported simultaneously                   |
| `--fastbcp_p`             | -4      | Parallel threads per large table                 |
| `--large_table_threshold` | 100000  | Row count below which tables export sequentially |

**Parallel degree guidelines** (`--fastbcp_p`):

* **< 100K rows**: -4 (default auto-scaled)
* **100K-10M rows**: 2-4
* **10M-100M rows**: 4-8
* **> 100M rows**: 8-16

## Per-Table CLI Override[​](#per-table-cli-override "Direct link to Per-Table CLI Override")

```
--fastbcp_table_config "[schema.]table:method:key_column:parallel_degree"
```

The schema prefix is optional. When omitted, the config applies to all schemas containing that table (a warning is logged). Multiple tables separated by semicolons. Empty fields use defaults.

```
# Schema-qualified (targets specific schema)
--fastbcp_table_config "dbo.lineitem:DataDriven:YEAR(l_shipdate):8;dbo.orders:Ctid::4"

# Without schema (applies to all schemas, with warning)
--fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4"

# Timepartition -- key column uses parenthesized format
--fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4"
```

## Per-Table Database Override[​](#per-table-database-override "Direct link to Per-Table Database Override")

Insert into the `partition_columns` table for persistent configuration:

```
INSERT INTO partition_columns (
    source_db_type, source_database, source_schema, source_table,
    fastbcp_method, fastbcp_parallel_degree, fastbcp_distribution_key
) VALUES (
    'postgres', 'analytics', 'public', 'events',
    'Ctid', 4, NULL
);
```

Database configuration takes priority over CLI configuration, which takes priority over auto-detection.

## Monitoring[​](#monitoring "Direct link to Monitoring")

```
SELECT source_table, status, finished_at - started_at AS duration, row_count
FROM jobs
WHERE run_id = 'your-run-id'
ORDER BY duration DESC;
```


---

# Incremental Sync

Sync only changed data using watermark-based incremental exports.

## What is Incremental Sync?[​](#what-is-incremental-sync "Direct link to What is Incremental Sync?")

LakeXpress tracks a "high watermark" (the highest value in a timestamp or numeric column) and only exports rows above that watermark on subsequent syncs, instead of exporting entire tables every time.

## When to Use Incremental Sync[​](#when-to-use-incremental-sync "Direct link to When to Use Incremental Sync")

Use incremental sync when:

* **Regularly updated tables**: Source tables receive frequent inserts or updates
* **Frequent syncs**: Exports run multiple times per day or on a schedule
* **Cost efficiency**: You want to minimize network and compute usage
* **Time-series data**: Tables have timestamp columns tracking record creation or modification

**Examples:**

* Daily sales order updates to a data lake
* Event log aggregation from production systems
* Time-series metrics collection
* Transaction processing pipelines

## Supported Column Types[​](#supported-column-types "Direct link to Supported Column Types")

| Type        | Format                      | Example              | Use Case                        |
| ----------- | --------------------------- | -------------------- | ------------------------------- |
| `date`      | YYYY-MM-DD                  | 2025-01-08           | Daily transactions, order dates |
| `datetime`  | YYYY-MM-DD HH:MM<!-- -->:SS | 2025-01-08 14:30:25  | Precise time tracking           |
| `timestamp` | Database timestamp type     | 2025-01-08T14:30:25Z | Creation/modification times     |
| `integer`   | Numeric sequence            | 1000001              | Monotonic ID columns, batch IDs |

## Configuration Syntax[​](#configuration-syntax "Direct link to Configuration Syntax")

Define incremental tables with `--incremental_table`:

```
./LakeXpress config create \
  ... \
  --incremental_table "schema.table:column:type"
```

### Basic Syntax[​](#basic-syntax "Direct link to Basic Syntax")

```
schema.table:column:type
```

**Parameters:**

* `schema.table` - Fully qualified table name
* `column` - Column to track for watermark (should be indexed)
* `type` - Column type: `date`, `datetime`, `timestamp`, or `integer`

**Example:**

```
--incremental_table "sales.orders:created_date:date"
```

### Multiple Tables[​](#multiple-tables "Direct link to Multiple Tables")

Repeat the `--incremental_table` flag for each table:

```
./LakeXpress config create \
  ... \
  --incremental_table "sales.orders:created_date:date" \
  --incremental_table "sales.returns:return_date:date"
```

### Advanced Syntax (Optional)[​](#advanced-syntax-optional "Direct link to Advanced Syntax (Optional)")

```
schema.table:column:type[:direction][@start_value][!strategy]
```

**Extended Parameters:**

* `:direction` - Include (`:i`, default) or exclude (`:e`)
* `@start_value` - Override the initial watermark value
* `!strategy` - Loading strategy: `append` (default) or `upsert`

**Examples:**

```
# Include direction (explicit)
--incremental_table "sales.orders:created_date:date:i"

# Exclude from incremental sync
--incremental_table "sales.returns:return_date:date:e"

# Set initial watermark
--incremental_table "sales.orders:created_date:date@2025-01-01"

# Use UPSERT/MERGE strategy (updates existing rows)
--incremental_table "sales.orders:created_date:date!upsert"

# Combined: direction + start value + upsert
--incremental_table "sales.orders:created_date:date:i@2025-01-01!upsert"
```

## Non-incremental Tables[​](#non-incremental-tables "Direct link to Non-incremental Tables")

Tables not configured with `--incremental_table` are fully exported on each sync. Useful for small dimension or reference tables.

```
./LakeXpress config create \
  ... \
  --incremental_table "fact.sales:sale_date:date" \
  --n_jobs 4
```

* `fact.sales` - Exports only new records since last watermark
* All other tables in the schema - Fully exported on each sync

## See Also[​](#see-also "Direct link to See Also")

* [Loading Strategies](/0.3/advanced/incremental/loading-strategies.md)Append vs upsert strategies
* [Watermarks](/0.3/advanced/incremental/watermarks.md)Watermark tracking, safety lag, querying, and resetting
* [Examples](/0.3/advanced/incremental/examples.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/0.3/advanced/incremental/troubleshooting.md)Common issues, performance tips, and advanced topics
* [Quick Start Guide](/0.3/guides/quickstart.md)First export walkthrough
* [CLI Reference](/0.3/reference/cli-reference.md)All available options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Real-world usage examples
* [Intermediate Storage](/0.3/configuration/storage/storage-overview.md)Cloud storage configuration


---

# Examples

## Complete Example[​](#complete-example "Direct link to Complete Example")

### Step 1: Create Configuration with Incremental Tables[​](#step-1-create-configuration-with-incremental-tables "Direct link to Step 1: Create Configuration with Incremental Tables")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id ds_04_pg \
  --source_db_name tpch \
  --source_schema_name tpch_1_incremental \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id aws_s3_01 \
  --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
  --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
  --incremental_safety_lag 3600 \
  --generate_metadata \
  --n_jobs 4 \
  --fastbcp_p 2
```

| Parameter                                                           | Meaning                                             |
| ------------------------------------------------------------------- | --------------------------------------------------- |
| `--incremental_table "tpch_1_incremental.orders:o_orderdate:date"`  | Track orders by `o_orderdate` column                |
| `--incremental_table "tpch_1_incremental.lineitem:l_shipdate:date"` | Track lineitem by `l_shipdate` column               |
| `--incremental_safety_lag 3600`                                     | Wait 1 hour after the watermark (handles late data) |
| `--generate_metadata`                                               | Generate CDM metadata for exported tables           |

### Step 2: Run First Sync[​](#step-2-run-first-sync "Direct link to Step 2: Run First Sync")

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --sync_id <SYNC_ID>
```

First sync behavior:

* Exports **all** rows from `orders` and `lineitem`
* Records the highest `o_orderdate` and `l_shipdate` as watermarks
* Non-incremental tables are fully exported
* Stores watermarks in the LakeXpress DB

### Step 3: Subsequent Syncs[​](#step-3-subsequent-syncs "Direct link to Step 3: Subsequent Syncs")

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --sync_id <SYNC_ID>
```

Subsequent sync behavior:

* Loads previous watermarks from the LakeXpress DB
* Exports only rows where `o_orderdate > previous_watermark - safety_lag`
* Exports only rows where `l_shipdate > previous_watermark - safety_lag`
* Updates watermarks with current highest values
* Non-incremental tables are fully exported again

## Real-World Scenarios[​](#real-world-scenarios "Direct link to Real-World Scenarios")

### Scenario 1: Daily Order Processing[​](#scenario-1-daily-order-processing "Direct link to Scenario 1: Daily Order Processing")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name ecommerce \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --incremental_table "public.orders:created_at:datetime" \
  --incremental_table "public.order_items:created_at:datetime" \
  --publish_target snowflake_prod \
  --n_jobs 4

# Run daily via cron
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID>
```

* Day 1: Exports 1,000,000 orders (full load)
* Day 2: Exports \~5,000 new orders (incremental)
* Day 3: Exports \~4,800 new orders (incremental)
* Other tables (customers, products) fully exported daily

### Scenario 2: Event Log Aggregation[​](#scenario-2-event-log-aggregation "Direct link to Scenario 2: Event Log Aggregation")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id source_postgres \
  --source_db_name analytics \
  --source_schema_name events \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id aws_s3_01 \
  --incremental_table "events.pageviews:event_time:timestamp" \
  --incremental_table "events.clicks:event_time:timestamp" \
  --incremental_table "events.conversions:event_time:timestamp" \
  --incremental_safety_lag 600 \
  --sub_path production/events \
  --n_jobs 8 \
  --fastbcp_p 4

# Run every 10 minutes
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --sync_id <SYNC_ID>
```

* Ingests events from multiple tables continuously
* 10-minute safety lag handles processing delays

### Scenario 3: Time-Series Metrics[​](#scenario-3-time-series-metrics "Direct link to Scenario 3: Time-Series Metrics")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --source_db_auth_id source_postgres \
  --source_db_name monitoring \
  --source_schema_name metrics \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id azure_01 \
  --incremental_table "metrics.cpu_usage:recorded_at:timestamp" \
  --incremental_table "metrics.memory_usage:recorded_at:timestamp" \
  --incremental_table "metrics.disk_io:recorded_at:timestamp" \
  --incremental_safety_lag 300 \
  --n_jobs 4 \
  --generate_metadata

# Run every 5 minutes
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --sync_id <SYNC_ID>
```

* High-frequency metric collection to Azure storage
* Each sync captures the last 5+ minutes of data

### Combining with Snowflake Publishing[​](#combining-with-snowflake-publishing "Direct link to Combining with Snowflake Publishing")

```
./LakeXpress config create \
  ... \
  --incremental_table "sales.orders:order_date:date" \
  --incremental_table "sales.returns:return_date:date" \
  --target_storage_id s3_01 \
  --publish_target snowflake_prod \
  --publish_method internal \
  --publish_schema_pattern "{schema}_incremental" \
  --n_jobs 4

./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id <LXDB_AUTH_ID> \
  --sync_id <SYNC_ID>
```

Creates Snowflake tables continuously updated with new data. Non-incremental tables are fully exported and published on each sync.

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/0.3/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Loading Strategies](/0.3/advanced/incremental/loading-strategies.md)Append vs upsert strategies
* [Watermarks](/0.3/advanced/incremental/watermarks.md)Watermark tracking and safety lag


---

# Loading Strategies

## Append Strategy (Default)[​](#append-strategy-default "Direct link to Append Strategy (Default)")

Inserts new rows into the target table. Suitable when:

* Your target handles duplicates (e.g., external tables)
* You're building an append-only data lake
* You dedup at query time

```
--incremental_table "events.pageviews:event_time:timestamp!append"
```

## Upsert Strategy (MERGE)[​](#upsert-strategy-merge "Direct link to Upsert Strategy (MERGE)")

Uses MERGE to update existing rows and insert new ones. Requires:

* Primary keys defined on the table (stored in the LakeXpress DB)
* A target platform supporting MERGE (Snowflake, Databricks, Fabric)
* Managed/internal tables (not external tables)

```
--incremental_table "sales.orders:updated_at:datetime!upsert"
```

**How it works:**

1. New data loads into a staging table
2. MERGE runs: matching rows are updated, new rows are inserted
3. Staging table is dropped

**Benefits:**

* No duplicate rows in target
* Handles both inserts and updates
* Target always reflects current state

**Requirements:**

* Tables must have primary key columns defined in the source
* Use `--publish_method internal` or `--publish_method managed`
* Primary keys are automatically detected from the source database schema

**Example with Snowflake:**

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name ecommerce \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --incremental_table "public.orders:updated_at:datetime!upsert" \
  --incremental_table "public.customers:updated_at:datetime!upsert" \
  --publish_target snowflake_prod \
  --publish_method internal \
  --n_jobs 4
```

**Example with Databricks:**

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --source_db_auth_id source_mssql \
  --source_db_name sales \
  --source_schema_name dbo \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --incremental_table "dbo.products:modified_date:datetime!upsert" \
  --publish_target databricks_unity \
  --databricks_table_type managed \
  --n_jobs 4
```

## Choosing a Strategy[​](#choosing-a-strategy "Direct link to Choosing a Strategy")

| Scenario                     | Recommended Strategy |
| ---------------------------- | -------------------- |
| Append-only event logs       | `append`             |
| CDC (Change Data Capture)    | `upsert`             |
| Transaction history          | `append`             |
| Customer/product master data | `upsert`             |
| Time-series metrics          | `append`             |
| Order status updates         | `upsert`             |

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/0.3/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Examples](/0.3/advanced/incremental/examples.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/0.3/advanced/incremental/troubleshooting.md)Handling duplicates and other common issues


---

# Troubleshooting

## Missing Recent Data[​](#missing-recent-data "Direct link to Missing Recent Data")

**Symptom:** Data added to source appears 2-3 syncs later.

**Solution:** Reduce `--incremental_safety_lag`:

```
# Current: 1-hour lag
--incremental_safety_lag 3600

# Reduced: 10-minute lag
--incremental_safety_lag 600
```

## Duplicate Rows After Resync[​](#duplicate-rows-after-resync "Direct link to Duplicate Rows After Resync")

**Symptom:** Running the same sync twice produces duplicates.

**Solutions:**

1. **Use upsert strategy** (recommended): Configure with `!upsert` to use MERGE:

```
--incremental_table "sales.orders:created_date:date!upsert"
```

2. **Deduplicate in queries**: If using append mode, use ROW\_NUMBER() or QUALIFY at query time.

3. **Use external tables**: External tables over Parquet files replace files on each sync rather than inserting rows.

## Watermark Not Advancing[​](#watermark-not-advancing "Direct link to Watermark Not Advancing")

**Symptom:** Watermark stays at the same value across syncs.

**Solution:** Verify the incremental column has new values:

```
SELECT MAX(created_at) FROM orders;
-- If unchanged, the watermark is correct
```

## Performance Considerations[​](#performance-considerations "Direct link to Performance Considerations")

### Column Selection[​](#column-selection "Direct link to Column Selection")

Choose the incremental column carefully:

```
-- GOOD: Indexed column, never updates
--incremental_table "sales.orders:order_id:integer"

-- GOOD: Updated on insert only
--incremental_table "sales.orders:created_date:date"

-- CAUTION: May need resync if updated
--incremental_table "sales.orders:updated_at:datetime"

-- AVOID: May miss updates
--incremental_table "sales.orders:order_amount:integer"
```

### Indexing[​](#indexing "Direct link to Indexing")

Create an index on the watermark column:

```
CREATE INDEX idx_orders_created_at ON orders(created_at);
```

### Combining Incremental and Full Exports[​](#combining-incremental-and-full-exports "Direct link to Combining Incremental and Full Exports")

```
./LakeXpress config create \
  ... \
  --incremental_table "fact.transactions:txn_date:date" \
  --incremental_safety_lag 3600 \
  --n_jobs 4 \
  --fastbcp_p 8
```

* Fact tables configured with `--incremental_table` sync incrementally
* All other tables in the schema are fully exported (useful for dimension tables)

## Multiple Watermark Columns[​](#multiple-watermark-columns "Direct link to Multiple Watermark Columns")

Track changes in multiple columns with separate configurations:

```
# Configuration 1: Track by create date
lakexpress config create \
  ... \
  --incremental_table "transactions:created_at:datetime"

# Configuration 2: Track by update date
lakexpress config create \
  ... \
  --incremental_table "transactions:updated_at:datetime"
```

Run both syncs to capture creates and updates separately.

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/0.3/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Loading Strategies](/0.3/advanced/incremental/loading-strategies.md)Append vs upsert strategies
* [Watermarks](/0.3/advanced/incremental/watermarks.md)Watermark tracking, safety lag, and resetting


---

# Watermarks

## Watermark Tracking[​](#watermark-tracking "Direct link to Watermark Tracking")

LakeXpress maintains watermarks in the LakeXpress DB:

### First Sync (Full Load)[​](#first-sync-full-load "Direct link to First Sync (Full Load)")

```
Table: orders
Column: o_orderdate
Watermark: NULL -> 2025-12-31 (highest date in table)
Records exported: 1,500,000
```

### Second Sync (Incremental)[​](#second-sync-incremental "Direct link to Second Sync (Incremental)")

```
Query: SELECT * FROM orders WHERE o_orderdate > 2025-12-24 - 1 hour
Expected records: 50,000 (new orders in last week)
Watermark updated: 2025-12-31 -> 2026-01-05
```

## Safety Lag[​](#safety-lag "Direct link to Safety Lag")

The `--incremental_safety_lag` parameter handles late-arriving data:

```
./LakeXpress config create \
  ... \
  --incremental_table "events.raw_events:event_timestamp:datetime" \
  --incremental_safety_lag 3600 \
  ...
```

* `--incremental_safety_lag INT` - Lag in seconds (default: 0)

**Example with 1-hour lag:**

```
Current time: 2025-01-08 14:00:00
Watermark: 2025-01-08 10:00:00
Query includes: WHERE event_timestamp > 2025-01-08 09:00:00
(1 hour before watermark)
```

**When to use:**

* Asynchronous systems with delayed writes
* Multi-region databases with replication lag
* Event streams with out-of-order processing
* Financial transactions with settlement delays

## Querying Watermarks[​](#querying-watermarks "Direct link to Querying Watermarks")

Inspect tracked watermarks by querying the LakeXpress DB:

```
-- View all incremental configurations
SELECT
    sync_id,
    config_name,
    source_table,
    incremental_column,
    last_watermark,
    updated_at
FROM sync_configurations
WHERE is_incremental = true
ORDER BY updated_at DESC;

-- View recent watermark updates
SELECT
    run_id,
    sync_id,
    source_table,
    previous_watermark,
    new_watermark,
    rows_exported,
    started_at,
    completed_at
FROM incremental_watermarks
ORDER BY completed_at DESC
LIMIT 10;
```

## Resetting Watermarks[​](#resetting-watermarks "Direct link to Resetting Watermarks")

To do a full reload:

```
# Option 1: Delete and recreate the configuration
./LakeXpress config delete \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id 20251208-xxxxx

# Then create a new one
./LakeXpress config create ...

# Option 2: Override watermark on next sync
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID> \
  --reset-watermarks
```

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/0.3/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Examples](/0.3/advanced/incremental/examples.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/0.3/advanced/incremental/troubleshooting.md)Fixing watermark issues and missing data


---

# Compatibility

## Supported Databases[​](#supported-databases "Direct link to Supported Databases")

| Database Type | As Source     | As LakeXpress DB |
| ------------- | ------------- | ---------------- |
| SQL Server    | Supported     | Supported        |
| PostgreSQL    | Supported     | Supported        |
| Oracle        | Supported     | Not supported    |
| MySQL         | Supported     | Supported        |
| MariaDB       | Supported     | Not supported    |
| SAP HANA      | Supported     | Not supported    |
| Teradata      | Supported     | Not supported    |
| SQLite        | Not supported | Supported        |
| DuckDB        | Not supported | Supported        |

## Supported Storage Backends[​](#supported-storage-backends "Direct link to Supported Storage Backends")

| Storage Backend                      | Support Status |
| ------------------------------------ | -------------- |
| **Local Filesystem**                 | Supported      |
| **AWS S3**                           | Supported      |
| **S3-Compatible (OVH, MinIO, etc.)** | Supported      |
| **Google Cloud Storage (GCS)**       | Supported      |
| **Azure Storage (ADLS Gen2, Blob)**  | Supported      |

## Supported Publishing Targets[​](#supported-publishing-targets "Direct link to Supported Publishing Targets")

After export, LakeXpress can create tables in analytics platforms as **external** (data stays in cloud storage) or **internal** (data loaded into native storage).

| Publishing Target    | External  | Internal      |
| -------------------- | --------- | ------------- |
| **Snowflake**        | Supported | Supported     |
| **Databricks**       | Supported | Supported     |
| **Microsoft Fabric** | Supported | Supported     |
| **BigQuery**         | Supported | Supported     |
| **MotherDuck**       | Supported | Supported     |
| **Amazon Redshift**  | Supported | Supported     |
| **AWS Glue**         | Supported | Not supported |
| **DuckLake**         | Supported | Not supported |

Set `--publish_method external` or `--publish_method internal`. Default is `external`.

Publishing guides

Setup, authentication and options for each platform

* [![](/img/logos/snowflake.svg)Snowflake](/0.3/integrations/snowflake-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue](/0.3/integrations/glue-publishing.md)
* [![](/img/logos/databricks.svg)Databricks](/0.3/integrations/databricks-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric](/0.3/integrations/fabric-publishing.md)
* [![](/img/logos/bigquery.svg)BigQuery](/0.3/integrations/bigquery-publishing.md)
* [Amazon Redshift](/0.3/integrations/redshift-publishing.md)
* [![](/img/logos/motherduck.svg)MotherDuck](/0.3/integrations/motherduck-publishing.md)
* [![](/img/logos/ducklake.svg)DuckLake](/0.3/integrations/ducklake-publishing.md)


---

# Database Connections

LakeXpress uses two distinct database roles, each requiring its own connection entry in the credentials file:

* **LakeXpress DB** — An internal database used by LakeXpress to store configuration, track sync state, and log pipeline runs. This is not a source of data; it is LakeXpress's own bookkeeping store. You point LakeXpress at it with `--lxdb_auth_id`.
* **Source Database** — The database you want to extract data from (e.g., a production PostgreSQL or Oracle instance). You point LakeXpress at it with `--source_db_auth_id`.

Both connections are defined in the same credentials JSON file, distinguished by the key name you give each entry.

## Supported Databases[​](#supported-databases "Direct link to Supported Databases")

### Source Databases[​](#source-databases "Direct link to Source Databases")

| Database                                                     | Support Status | Notes                                             |
| ------------------------------------------------------------ | -------------- | ------------------------------------------------- |
| **[PostgreSQL](/0.3/configuration/databases/postgresql.md)** | Supported      | Ctid-based parallel export                        |
| **[Oracle](/0.3/configuration/databases/oracle.md)**         | Supported      | Thin and thick modes, Rowid-based parallel export |
| **[SQL Server](/0.3/configuration/databases/sql-server.md)** | Supported      | Physloc-based parallel export                     |
| **[MySQL](/0.3/configuration/databases/mysql.md)**           | Supported      | Partition detection                               |
| **[MariaDB](/0.3/configuration/databases/mariadb.md)**       | Supported      | Dedicated `mariadb` ds\_type                      |
| **[Teradata](/0.3/configuration/databases/teradata.md)**     | Supported      | Requires `teradatasql` driver                     |
| **[SAP HANA](/0.3/configuration/databases/sap-hana.md)**     | Supported      | Requires `hdbcli` and HDBDOTNETCORE for FastBCP   |

### LakeXpress DB[​](#lakexpress-db "Direct link to LakeXpress DB")

| Database                                                     | Support Status | Notes                                                        |
| ------------------------------------------------------------ | -------------- | ------------------------------------------------------------ |
| **[PostgreSQL](/0.3/configuration/databases/postgresql.md)** | Supported      | Recommended for production use                               |
| **[SQL Server](/0.3/configuration/databases/sql-server.md)** | Supported      |                                                              |
| **[MySQL](/0.3/configuration/databases/mysql.md)**           | Supported      |                                                              |
| **[SQLite](/0.3/configuration/databases/sqlite.md)**         | Supported      | Good for single-user and development use; no server required |
| **[DuckDB](/0.3/configuration/databases/duckdb.md)**         | Supported      | Good for single-user and development use; no server required |

## Credentials File Format[​](#credentials-file-format "Direct link to Credentials File Format")

All database credentials are stored in a single JSON file (typically named `auth.json`). Each top-level key is a connection identifier you choose — you reference these identifiers on the command line via `--lxdb_auth_id` and `--source_db_auth_id`. The file can hold as many connection entries as you need.

Create a JSON file with your database credentials:

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_oracle": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-server.com",
      "port": 1521,
      "database": "orclpdb1",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  },
  "lxdb_sqlite": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/path/to/lakexpress_log.sqlite"
    }
  }
}
```

## Command-Line Examples[​](#command-line-examples "Direct link to Command-Line Examples")

### Passing the Credentials File[​](#passing-the-credentials-file "Direct link to Passing the Credentials File")

```
# PostgreSQL source, PostgreSQL LakeXpress DB
./LakeXpress -a auth.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres

# Oracle source, SQL Server LakeXpress DB
./LakeXpress -a auth.json \
  --lxdb_auth_id lxdb_mssql \
  --source_db_auth_id source_oracle

# SQL Server source, SQLite LakeXpress DB
./LakeXpress -a auth.json \
  --lxdb_auth_id lxdb_sqlite \
  --source_db_auth_id source_mssql

# MySQL source, MySQL LakeXpress DB
./LakeXpress -a auth.json \
  --lxdb_auth_id lxdb_mysql \
  --source_db_auth_id source_mysql

# MySQL source, DuckDB LakeXpress DB
./LakeXpress -a auth.json \
  --lxdb_auth_id lxdb_duckdb \
  --source_db_auth_id source_mysql
```

## Publishing Credentials[​](#publishing-credentials "Direct link to Publishing Credentials")

Publishing targets use their own credential entries in the same `credentials.json` file. See each publishing guide for authentication setup:

* [Snowflake](/0.3/integrations/snowflake-publishing.md)
* [Databricks](/0.3/integrations/databricks-publishing.md)
* [Microsoft Fabric](/0.3/integrations/fabric-publishing.md)
* [BigQuery](/0.3/integrations/bigquery-publishing.md)
* [MotherDuck](/0.3/integrations/motherduck-publishing.md)
* [AWS Glue](/0.3/integrations/glue-publishing.md)

## Security Best Practices[​](#security-best-practices "Direct link to Security Best Practices")

### 1. Credential Storage[​](#1-credential-storage "Direct link to 1. Credential Storage")

* Never commit credentials to version control
* Restrict `auth.json` permissions:
  <!-- -->
  ```
  chmod 600 auth.json
  ```
* Use environment-specific auth files (dev, staging, prod)

### 2. Least Privilege[​](#2-least-privilege "Direct link to 2. Least Privilege")

* Create dedicated database users for LakeXpress
* Grant only `SELECT` on source databases — LakeXpress never writes to them
* Grant full privileges on the LakeXpress DB — LakeXpress needs to create and update its own tracking tables there

### 3. Network Security[​](#3-network-security "Direct link to 3. Network Security")

* Use SSL/TLS connections
* Restrict database access by IP
* Use VPN or private networks

### 4. Password Management[​](#4-password-management "Direct link to 4. Password Management")

* Use strong passwords (12+ characters, mixed case, numbers, symbols)
* Rotate passwords regularly
* Consider secret managers (AWS Secrets Manager, HashiCorp Vault)

### 5. Environment Variables for Credentials[​](#5-environment-variables-for-credentials "Direct link to 5. Environment Variables for Credentials")

LakeXpress supports `$env{VAR_NAME}` syntax in any string value in the credentials JSON. An error is raised if a referenced variable is not set.

```
{
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  }
}
```

Set variables before running LakeXpress:

```
# Linux
export LX_PG_USER="dataexport"
export LX_PG_PASSWORD="SecureP@ssw0rd"
export LX_MSSQL_USER="sa"
export LX_MSSQL_PASSWORD="StrongP@ssw0rd"

# Windows (cmd)
set LX_PG_USER=dataexport
set LX_PG_PASSWORD=SecureP@ssw0rd

# Windows (PowerShell)
$env:LX_PG_USER = "dataexport"
$env:LX_PG_PASSWORD = "SecureP@ssw0rd"
```

Plain-text values still work -- values without `$env{` are returned as-is.

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/0.3/guides/quickstart.md)Getting started
* [CLI Reference](/0.3/reference/cli-reference.md)Command-line options
* [Intermediate Storage](/0.3/configuration/storage/storage-overview.md)Storage configuration


---

# DuckDB (LakeXpress DB Only)

DuckDB can be used as the LakeXpress DB — the internal database where LakeXpress stores its configuration, tracks sync state, and logs pipeline runs. It is not supported as a source database.

Like SQLite, DuckDB is file-based and requires no server. Only file-based databases are supported; in-memory mode (`:memory:`) is not supported because LakeXpress needs its tracking data to persist between runs.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                  |
| ----------- | -------- | ---------------------------- |
| `ds_type`   | Yes      | Must be `"duckdb"`           |
| `auth_mode` | Yes      | Must be `"filesystem"`       |
| `filepath`  | Yes      | Path to DuckDB database file |

## Example[​](#example "Direct link to Example")

```
{
  "lxdb_duckdb": {
    "ds_type": "duckdb",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/var/lib/lakexpress/export_log.duckdb"
    }
  }
}
```

## When to Use DuckDB[​](#when-to-use-duckdb "Direct link to When to Use DuckDB")

**Good for:** single-user setups, development, testing, fast analytical queries, lightweight embedded needs.

**Avoid for:** concurrent writes, high-concurrency production, distributed systems.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Unable to open database file:** Check file path, verify directory exists with write permissions, use absolute paths, check disk space.

**Database is locked:** Ensure no other process is using the file, close existing connections, check for stale lock files.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# MariaDB

MariaDB is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Although MariaDB is derived from MySQL, use `ds_type: "mariadb"` rather than `"mysql"`. LakeXpress has a dedicated driver for MariaDB to handle compatibility differences between the two.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description          |
| ----------- | -------- | -------------------- |
| `ds_type`   | Yes      | Must be `"mariadb"`  |
| `auth_mode` | Yes      | Must be `"classic"`  |
| `server`    | Yes      | Hostname or IP       |
| `port`      | Yes      | Port (default: 3306) |
| `database`  | Yes      | Database name        |
| `username`  | Yes      | MariaDB username     |
| `password`  | Yes      | MariaDB password     |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON your_database.* TO 'lakexpress_user'@'%';

-- INFORMATION_SCHEMA access is granted by default
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_mariadb": {
    "ds_type": "mariadb",
    "auth_mode": "classic",
    "info": {
      "server": "localhost",
      "port": 3306,
      "database": "your-database",
      "username": "$env{LX_MARIADB_USER}",
      "password": "$env{LX_MARIADB_PASSWORD}"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Access denied for user:** Verify credentials, check host access: `SELECT user, host FROM mysql.user WHERE user='your_user';`, grant if needed: `GRANT ALL PRIVILEGES ON database.* TO 'user'@'%';`

**Unknown database:** Verify name: `SHOW DATABASES;`, create if needed: `CREATE DATABASE your_database;`

**Connection refused:** Check MariaDB is running (`systemctl status mariadb`), verify port (`netstat -tlnp | grep 3306`), check `bind-address` allows remote connections, check firewall (`sudo ufw allow 3306/tcp`).

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# MySQL

MySQL can be used as a **source database** (the database LakeXpress extracts data from) and as the **LakeXpress DB** (the internal database LakeXpress uses for configuration, sync tracking, and pipeline logging). Both roles use the same connection parameter format.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description          |
| ----------- | -------- | -------------------- |
| `ds_type`   | Yes      | Must be `"mysql"`    |
| `auth_mode` | Yes      | Must be `"classic"`  |
| `server`    | Yes      | Hostname or IP       |
| `port`      | Yes      | Port (default: 3306) |
| `database`  | Yes      | Database name        |
| `username`  | Yes      | MySQL username       |
| `password`  | Yes      | MySQL password       |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON your_database.* TO 'lakexpress_user'@'%';

-- LakeXpress DB user:
GRANT ALL PRIVILEGES ON lakexpress_log.* TO 'lakexpress_user'@'%';

-- INFORMATION_SCHEMA access is granted by default
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_mysql": {
    "ds_type": "mysql",
    "auth_mode": "classic",
    "info": {
      "server": "localhost",
      "port": 3306,
      "database": "your-database",
      "username": "$env{LX_MYSQL_USER}",
      "password": "$env{LX_MYSQL_PASSWORD}"
    }
  }
}
```

## Example: LakeXpress DB[​](#example-lakexpress-db "Direct link to Example: LakeXpress DB")

```
{
  "lxdb_mysql": {
    "ds_type": "mysql",
    "auth_mode": "classic",
    "info": {
      "server": "localhost",
      "port": 3306,
      "database": "lakexpress_log",
      "username": "$env{LX_MYSQL_USER}",
      "password": "$env{LX_MYSQL_PASSWORD}"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Access denied for user:** Verify credentials, check host access: `SELECT user, host FROM mysql.user WHERE user='your_user';`, grant if needed: `GRANT ALL PRIVILEGES ON database.* TO 'user'@'%';`

**Unknown database:** Verify name: `SHOW DATABASES;`, create if needed: `CREATE DATABASE your_database;`

**Connection refused:** Check MySQL is running (`systemctl status mysql`), verify port (`netstat -tlnp | grep 3306`), check `bind-address` allows remote connections, check firewall (`sudo ufw allow 3306/tcp`).

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# Oracle

Oracle is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                           |
| ----------- | -------- | ------------------------------------- |
| `ds_type`   | Yes      | Must be `"oracle"`                    |
| `auth_mode` | Yes      | Must be `"classic"`                   |
| `username`  | Yes      | Oracle username                       |
| `password`  | Yes      | Oracle password                       |
| `server`    | Yes      | Hostname or IP                        |
| `port`      | Yes      | Port (default: 1521)                  |
| `database`  | Yes      | Service name or SID                   |
| `lib_dir`   | No       | Client library directory (thick mode) |

## Thin vs Thick Mode[​](#thin-vs-thick-mode "Direct link to Thin vs Thick Mode")

**Thin Mode (Recommended):**

* No client libraries required
* Oracle 12.1+
* Omit `lib_dir`

**Thick Mode:**

* Required for Oracle 11.2 and earlier
* Requires Oracle Instant Client
* Set `lib_dir` to the Instant Client directory

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

LakeXpress uses RowID-based parallel exports to read Oracle tables efficiently. The database user needs catalog access for metadata discovery (table structure, RowID ranges, and export method selection):

```
-- As Oracle DBA (e.g., sys as sysdba)
GRANT SELECT_CATALOG_ROLE TO your_username;
```

## Example: Thin Mode[​](#example-thin-mode "Direct link to Example: Thin Mode")

```
{
  "oracle_prod": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-prod.company.com",
      "port": 1521,
      "database": "PRODPDB"
    }
  }
}
```

## Example: Thick Mode[​](#example-thick-mode "Direct link to Example: Thick Mode")

```
{
  "oracle_legacy": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-11g.company.com",
      "port": 1521,
      "database": "LEGACY",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**ORA-12154: TNS could not resolve connect identifier:** Verify service name, check `tnsnames.ora`, try IP instead of hostname, confirm listener is running.

**ORA-28000: account is locked:**

```
ALTER USER your_username ACCOUNT UNLOCK;
```

**Thick mode library not found:** Verify `lib_dir` path, confirm Instant Client is installed, check file permissions, set `LD_LIBRARY_PATH` on Linux.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# PostgreSQL

PostgreSQL can be used as a **source database** (the database LakeXpress extracts data from) and as the **LakeXpress DB** (the internal database LakeXpress uses for configuration, sync tracking, and pipeline logging). Both roles use the same connection parameter format.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description          |
| ----------- | -------- | -------------------- |
| `ds_type`   | Yes      | Must be `"postgres"` |
| `auth_mode` | Yes      | Must be `"classic"`  |
| `username`  | Yes      | PostgreSQL username  |
| `password`  | Yes      | PostgreSQL password  |
| `server`    | Yes      | Hostname or IP       |
| `port`      | Yes      | Port (default: 5432) |
| `database`  | Yes      | Database name        |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

Permissions differ depending on how you are using PostgreSQL:

```
-- When used as a source database (read-only access):
GRANT SELECT ON ALL TABLES IN SCHEMA public TO lakexpress_user;
GRANT USAGE ON SCHEMA public TO lakexpress_user;

-- information_schema access is granted by default

-- When used as the LakeXpress DB (LakeXpress manages its own tables here):
GRANT ALL PRIVILEGES ON DATABASE lakexpress_log TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "postgres_prod": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "pg-prod.company.com",
      "port": 5432,
      "database": "analytics"
    }
  }
}
```

## Example: LakeXpress DB[​](#example-lakexpress-db "Direct link to Example: LakeXpress DB")

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "pg-prod.company.com",
      "port": 5432,
      "database": "lakexpress_log"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Connection refused:** Check PostgreSQL is running (`systemctl status postgresql`), verify `pg_hba.conf` allows your IP, check firewall (`sudo ufw allow 5432/tcp`).

**Authentication failed:** Verify credentials, check password encryption in `pg_hba.conf`, confirm user exists: `SELECT * FROM pg_user WHERE usename='your_user';`

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# SAP HANA

SAP HANA is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Connecting to SAP HANA requires the `hdbcli` Python driver. If you are using FastBCP for high-throughput exports, the HDBDOTNETCORE runtime is also required (see the note at the bottom of this page).

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                                                                                      |
| ----------- | -------- | ------------------------------------------------------------------------------------------------ |
| `ds_type`   | Yes      | Must be `"saphana"`                                                                              |
| `auth_mode` | Yes      | `"classic"` for individual fields; `"connection_string"` to supply a full ODBC connection string |
| `server`    | Yes      | Hostname or IP                                                                                   |
| `port`      | Yes      | Port (default: 39017)                                                                            |
| `database`  | Yes      | Database name                                                                                    |
| `username`  | Yes      | SAP HANA username                                                                                |
| `password`  | Yes      | SAP HANA password                                                                                |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON SCHEMA your_schema TO lakexpress_user;

-- Metadata access:
GRANT SELECT ON SYS.TABLES TO lakexpress_user;
GRANT SELECT ON SYS.COLUMNS TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_saphana": {
    "ds_type": "saphana",
    "auth_mode": "classic",
    "info": {
      "server": "hana-prod.company.com",
      "port": 39017,
      "database": "HDB",
      "username": "$env{LX_HANA_USER}",
      "password": "$env{LX_HANA_PASSWORD}"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

Use `auth_mode: "connection_string"` when you need to pass advanced ODBC driver options, or when your environment already manages connection strings centrally. The entire connection is expressed as a single `connection_string` value; the individual `server`, `port`, `database`, `username`, and `password` fields are not used.

```
{
  "source_saphana_cs": {
    "ds_type": "saphana",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "DRIVER={HDBODBC};SERVERNODE=$env{LX_HANA_HOST}:39017;DATABASE=$env{LX_HANA_DB};UID=$env{LX_HANA_USER};PWD=$env{LX_HANA_PASSWORD}"
    }
  }
}
```

note

FastBCP requires the HDBDOTNETCORE runtime. Set the `HDBDOTNETCORE` environment variable to the runtime directory path before running LakeXpress.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# SQL Server

SQL Server can be used as a **source database** (the database LakeXpress extracts data from) and as the **LakeXpress DB** (the internal database LakeXpress uses for configuration, sync tracking, and pipeline logging). Both roles use the same connection parameter format.

note

LakeXpress uses the ODBC driver for SQL Server connections. If ODBC drivers are not installed on the host machine, LakeXpress automatically falls back to PyMSSQL.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description          |
| ----------- | -------- | -------------------- |
| `ds_type`   | Yes      | Must be `"mssql"`    |
| `auth_mode` | Yes      | Must be `"classic"`  |
| `username`  | Yes      | SQL Server username  |
| `password`  | Yes      | SQL Server password  |
| `server`    | Yes      | Hostname or IP       |
| `port`      | Yes      | Port (default: 1433) |
| `database`  | Yes      | Database name        |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

Permissions differ depending on how you are using SQL Server:

```
-- When used as a source database (read-only access):
GRANT SELECT ON SCHEMA::dbo TO lakexpress_user;
GRANT VIEW DEFINITION ON SCHEMA::dbo TO lakexpress_user;

-- INFORMATION_SCHEMA access is granted by default

-- When used as the LakeXpress DB (LakeXpress manages its own tables here):
GRANT CONTROL ON DATABASE::lakexpress_log TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "mssql_prod": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-prod.company.com",
      "port": 1433,
      "database": "SalesDB"
    }
  }
}
```

## Example: LakeXpress DB[​](#example-lakexpress-db "Direct link to Example: LakeXpress DB")

```
{
  "lxdb_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "mssql-prod.company.com",
      "port": 1433,
      "database": "lakexpress_log"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Login failed for user:** Confirm SQL Server authentication is enabled (not Windows-only), verify user exists with correct permissions, check password and IP restrictions.

**Cannot open database requested by the login:** Verify database name, check user access, confirm database is online: `SELECT state_desc FROM sys.databases WHERE name='YourDB'`

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# SQLite (LakeXpress DB Only)

SQLite can be used as the LakeXpress DB — the internal database where LakeXpress stores its configuration, tracks sync state, and logs pipeline runs. It is not supported as a source database.

Because SQLite is file-based and requires no server, it is the simplest option to get started with.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                  |
| ----------- | -------- | ---------------------------- |
| `ds_type`   | Yes      | Must be `"sqlite"`           |
| `auth_mode` | Yes      | Must be `"filesystem"`       |
| `filepath`  | Yes      | Path to SQLite database file |

## Example[​](#example "Direct link to Example")

```
{
  "lxdb_local": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/var/lib/lakexpress/export_log.sqlite"
    }
  }
}
```

## When to Use SQLite[​](#when-to-use-sqlite "Direct link to When to Use SQLite")

**Good for:** single-user setups, development, testing, deployments without a database server.

**Avoid for:** multi-user environments, high-concurrency production, distributed systems.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Unable to open database file:** Check file path, verify directory exists with write permissions, use absolute paths, check disk space.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# Teradata

Teradata is supported as a **source database** -- the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Connecting to Teradata requires the `teradatasql` Python driver.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                                                                                 |
| ----------- | -------- | ------------------------------------------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"teradata"`                                                                        |
| `auth_mode` | Yes      | `"classic"` for individual fields; `"connection_string"` to supply a full connection string |
| `server`    | Yes      | Hostname or IP                                                                              |
| `port`      | No       | Port (default: 1025)                                                                        |
| `database`  | Yes      | Database name                                                                               |
| `username`  | Yes      | Teradata username                                                                           |
| `password`  | Yes      | Teradata password                                                                           |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON your_database TO lakexpress_user;

-- Metadata access (table discovery, column metadata, primary keys, foreign keys):
GRANT SELECT ON DBC.TablesV TO lakexpress_user;
GRANT SELECT ON DBC.ColumnsV TO lakexpress_user;
GRANT SELECT ON DBC.IndicesV TO lakexpress_user;
GRANT SELECT ON DBC.All_RI_ChildrenV TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_teradata": {
    "ds_type": "teradata",
    "auth_mode": "classic",
    "info": {
      "server": "teradata-prod.company.com",
      "port": 1025,
      "database": "DW",
      "username": "$env{LX_TD_USER}",
      "password": "$env{LX_TD_PASSWORD}"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

Use `auth_mode: "connection_string"` when you need to pass advanced driver options.

```
{
  "source_teradata_cs": {
    "ds_type": "teradata",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "{\"host\":\"$env{LX_TD_HOST}\",\"dbs_port\":\"1025\",\"user\":\"$env{LX_TD_USER}\",\"password\":\"$env{LX_TD_PASSWORD}\",\"database\":\"DW\"}"
    }
  }
}
```

## FastBCP Methods[​](#fastbcp-methods "Direct link to FastBCP Methods")

All generic FastBCP distribution methods are supported with Teradata sources:

| Method          | Description                                                                                                           |
| --------------- | --------------------------------------------------------------------------------------------------------------------- |
| `None`          | Single-threaded export (default)                                                                                      |
| `Random`        | Parallel export using an integer column                                                                               |
| `DataDriven`    | Split by column values into separate files                                                                            |
| `RangeId`       | Split by column min/max ranges                                                                                        |
| `Ntile`         | Evenly distributed chunks                                                                                             |
| `Timepartition` | Partition by date/time column -- supports `(col,year)`, `(col,year,month)`, `(col,year,month,day)`, `(col,year,week)` |

Timepartition splits exports into Hive-style paths (e.g., `year=2008/month=01/table.parquet`) where each parallel thread handles a specific time period. It is well suited for large historical tables with a date column.

```
# Single table
--fastbcp_table_config "ORDERS:Timepartition:(ORDER_DATE,year,month):8"

# Multiple tables with different date columns
--fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4" \
--fastbcp_table_config "PRESCRIPTION_DRUG_EVENTS:Timepartition:(SRVC_DT,year,month):4"
```

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/0.3/configuration/database-configuration.md)Auth file format, security best practices


---

# Azure Storage

Supports Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage via Service Principal authentication.

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "azure_storage": {
    "ds_type": "azure",
    "auth_mode": "service_principal",
    "info": {
      "directory": "abfss://your-container@your-storage-account.dfs.core.windows.net/path/to/exports",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}
```

note

Use `$env{VAR_NAME}` syntax to reference environment variables. Set the variable before running LakeXpress (e.g., `export LX_AZURE_CLIENT_SECRET="your-secret"`).

| Field                 | Required | Description                                                     |
| --------------------- | -------- | --------------------------------------------------------------- |
| `ds_type`             | Yes      | Must be `"azure"`                                               |
| `auth_mode`           | Yes      | Must be `"service_principal"`                                   |
| `directory`           | Yes      | Azure storage URL (`abfss://` for ADLS Gen2, `abs://` for Blob) |
| `azure_client_id`     | Yes      | Application (client) ID from Azure AD app registration          |
| `azure_tenant_id`     | Yes      | Directory (tenant) ID from Azure AD                             |
| `azure_client_secret` | Yes      | Client secret value from Azure AD app                           |

## Storage URL Formats[​](#storage-url-formats "Direct link to Storage URL Formats")

**ADLS Gen2:**

```
abfss://container-name.dfs.core.windows.net/path/to/exports
```

**Azure Blob Storage:**

```
abs://container-name.blob.core.windows.net/path/to/exports
```

The URL format uses `.dfs.core.windows.net` (not `@storageaccount.dfs.core.windows.net`).

## Azure Authentication Setup[​](#azure-authentication-setup "Direct link to Azure Authentication Setup")

**1. Create an Azure AD Application (Service Principal):**

In [Azure Portal App Registrations](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps):

* Click "New registration"
* Note the **Application (client) ID** and **Directory (tenant) ID**
* Go to "Certificates & secrets" -> "New client secret"
* Copy the **client secret value** (shown only once)

**2. Assign Storage Permissions:**

In your Storage Account -> Access Control (IAM), add the **"Storage Blob Data Contributor"** role to your service principal.

[Azure RBAC roles reference](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor)

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id azure_01 \
        --fastbcp_dir_path /path/to/fastbcp
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

**ADLS Gen2:**

```
abfss://container@storage.dfs.core.windows.net/path/to/exports/schema_name/table_name/part-00000.parquet
```

**With `--sub_path`:**

```
abfss://container@storage.dfs.core.windows.net/path/to/exports/sub_path/schema_name/table_name/part-00000.parquet
```

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

The Service Principal needs **Storage Blob Data Contributor**, which grants:

* `Microsoft.Storage/storageAccounts/blobServices/containers/read`
* `Microsoft.Storage/storageAccounts/blobServices/containers/write`
* `Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action`

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem                        | Solution                                                                                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Authentication / access denied | Verify client ID, tenant ID, and secret. Confirm "Storage Blob Data Contributor" role. Check network access rules. Verify container exists. |
| Invalid URL format             | Use `abfss://` for ADLS Gen2, `abs://` for Blob. Format: `abfss://container@storageaccount.dfs.core.windows.net/path`.                      |
| Connection timeout             | Check firewall rules, storage account firewall allowlist, and private endpoint restrictions.                                                |


---

# Google Cloud Storage (GCS)

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "gcs_storage": {
    "ds_type": "gcs",
    "auth_mode": "profile",
    "info": {
      "directory": "gs://your-bucket-name/path/to/exports",
      "profile": "/path/to/service-account-key.json"
    }
  }
}
```

| Field       | Required | Description                               |
| ----------- | -------- | ----------------------------------------- |
| `ds_type`   | Yes      | Must be `"gcs"`                           |
| `auth_mode` | Yes      | Must be `"profile"`                       |
| `directory` | Yes      | Full GCS URL (e.g., `gs://bucket/path`)   |
| `profile`   | Yes      | Path to GCS service account JSON key file |

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id gcs_01 \
        --fastbcp_dir_path /path/to/fastbcp
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
gs://bucket-name/base_path/schema_name/table_name/part-00000.parquet
```

## Service Account Permissions[​](#service-account-permissions "Direct link to Service Account Permissions")

Required roles:

* `Storage Object Creator`
* `Storage Object Viewer` (if reading is needed)

Or custom IAM permissions:

* `storage.objects.create`
* `storage.objects.delete` (if overwriting)
* `storage.buckets.get`

## Creating a Service Account[​](#creating-a-service-account "Direct link to Creating a Service Account")

```
gcloud iam service-accounts create lakexpress-export \
    --display-name="LakeXpress Export Service Account"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectCreator"

gcloud iam service-accounts keys create service-account-key.json \
    --iam-account=lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem               | Solution                                                                                      |
| --------------------- | --------------------------------------------------------------------------------------------- |
| Authentication errors | Verify key file path. Check `Storage Object Creator` role. Ensure service account is enabled. |
| Quota exceeded        | Check GCS quotas in Google Cloud Console. Request increase if needed.                         |


---

# Local Filesystem

## Configuration[​](#configuration "Direct link to Configuration")

```
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --output_dir ./exports \
        --fastbcp_dir_path /path/to/fastbcp
```

## Directory Structure[​](#directory-structure "Direct link to Directory Structure")

```
output_dir/
└── schema_name/
    └── table_name/
        ├── part-00000.parquet
        ├── part-00001.parquet
        └── ...
```

With `--sub_path staging/temp`:

```
exports/
└── staging/
    └── temp/
        └── schema_name/
            └── table_name/
                ├── part-00000.parquet
                └── ...
```

## Permissions[​](#permissions "Direct link to Permissions")

The user running LakeXpress needs write access to the output directory, sufficient disk space, and the ability to create subdirectories.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem           | Solution                                                          |
| ----------------- | ----------------------------------------------------------------- |
| Permission denied | `chmod 755 /path/to/output_dir`. Ensure parent directories exist. |


---

# Microsoft Fabric OneLake

Exports Parquet files to OneLake via Service Principal authentication.

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "onelake_storage": {
    "ds_type": "onelake",
    "auth_mode": "service_principal",
    "info": {
      "directory": "onelake://your-workspace-name/your-lakehouse-name/",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}
```

| Field                 | Required | Description                           |
| --------------------- | -------- | ------------------------------------- |
| `ds_type`             | Yes      | Must be `"onelake"`                   |
| `auth_mode`           | Yes      | Must be `"service_principal"`         |
| `directory`           | Yes      | `onelake://workspace/lakehouse/path/` |
| `azure_client_id`     | Yes      | Application (client) ID from Azure AD |
| `azure_tenant_id`     | Yes      | Directory (tenant) ID from Azure AD   |
| `azure_client_secret` | Yes      | Client secret value from Azure AD app |

## URL Format[​](#url-format "Direct link to URL Format")

```
onelake://workspace-name/lakehouse-name/path/to/exports/
```

Example:

```
onelake://FABRICPOC/lakexpress_lakehouse/exports/
```

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id fabric_onelake \
        --fastbcp_dir_path /path/to/fastbcp
```

With config creation:

```
python lxpress.py config create \
       -a auth.json \
       --lxdb_auth_id lxdb_ms \
       --source_db_auth_id ds_04_pg \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_dir_path /path/to/fastbcp \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id fabric_onelake \
       --generate_metadata
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
onelake://workspace/lakehouse/base_path/schema_name/table_name/part-00000.parquet
```

With `--sub_path`:

```
onelake://workspace/lakehouse/base_path/sub_path/schema_name/table_name/part-00000.parquet
```

## Authentication Setup[​](#authentication-setup "Direct link to Authentication Setup")

**1. Create an Azure AD Application (Service Principal):**

In [Azure Portal App Registrations](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps):

* Click "New registration"
* Note the **Application (client) ID** and **Directory (tenant) ID**
* Go to "Certificates & secrets" -> "New client secret"
* Copy the **client secret value** (shown only once)

**2. Grant OneLake Access in Microsoft Fabric:**

In the Fabric portal, go to Workspace settings -> "Manage access" and add the service principal with Contributor or Admin permissions. For finer control, grant access at the Lakehouse level.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem                        | Solution                                                                                                     |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| Authentication / access denied | Verify client ID, tenant ID, and secret. Confirm Contributor role on workspace. Verify OneLake is enabled.   |
| Invalid URL format             | Use `onelake://` prefix. Format: `onelake://workspace-name/lakehouse-name/path/`. Ensure path ends with `/`. |
| Connection timeout             | Check firewall rules and verify Fabric workspace is network-accessible.                                      |


---

# S3-Compatible Storage

All S3-compatible providers use `ds_type: "s3"`. Provider differentiation is handled through `endpoint_url` in the AWS profile.

Supported providers: Amazon S3, OVH S3, MinIO, Wasabi, DigitalOcean Spaces, Backblaze B2, and any S3-compatible object storage.

## Method 1: AWS Profile (Recommended for AWS)[​](#method-1-aws-profile-recommended-for-aws "Direct link to Method 1: AWS Profile (Recommended for AWS)")

Uses `~/.aws/config` and `~/.aws/credentials`:

```
{
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-bucket-name/lakexpress/exports",
      "profile": "your-aws-profile"
    }
  }
}
```

**`~/.aws/config`:**

```
[profile your-aws-profile]
region = us-east-1
```

**`~/.aws/credentials`:**

```
[your-aws-profile]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
```

| Field       | Required | Description                                |
| ----------- | -------- | ------------------------------------------ |
| `ds_type`   | Yes      | Must be `"s3"`                             |
| `auth_mode` | Yes      | Must be `"profile"`                        |
| `directory` | Yes      | Full S3 URL (e.g., `s3://bucket/path`)     |
| `profile`   | Yes      | AWS profile name from `~/.aws/credentials` |

***

## Method 2: S3-Compatible Providers (OVH, MinIO, etc.)[​](#method-2-s3-compatible-providers-ovh-minio-etc "Direct link to Method 2: S3-Compatible Providers (OVH, MinIO, etc.)")

Use an AWS profile with a custom `endpoint_url`.

**OVH S3:**

```
{
  "s3_02": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-ovh-bucket/lakexpress",
      "profile": "ovh"
    }
  }
}
```

```
[profile ovh]
endpoint_url = https://s3.gra.io.cloud.ovh.net
region = gra
```

**MinIO:**

```
{
  "s3_03": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://data-lake/exports",
      "profile": "minio"
    }
  }
}
```

```
[profile minio]
endpoint_url = http://localhost:9000
region = us-east-1
```

| Field       | Required | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"s3"`                                            |
| `auth_mode` | Yes      | Must be `"profile"`                                       |
| `directory` | Yes      | Full S3 URL (e.g., `s3://bucket/path`)                    |
| `profile`   | Yes      | AWS profile name (with `endpoint_url` in `~/.aws/config`) |

***

## Usage[​](#usage "Direct link to Usage")

```
# AWS S3
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_01 \
        --fastbcp_dir_path /path/to/fastbcp

# OVH S3
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_02 \
        --fastbcp_dir_path /path/to/fastbcp

# MinIO
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_03 \
        --fastbcp_dir_path /path/to/fastbcp
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
s3://bucket-name/base_path/schema_name/table_name/part-00000.parquet
```

With `--sub_path`:

```
s3://bucket-name/base_path/sub_path/schema_name/table_name/part-00000.parquet
```

## Required IAM Permissions (AWS S3)[​](#required-iam-permissions-aws-s3 "Direct link to Required IAM Permissions (AWS S3)")

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:s3:::your-bucket-name"
      ]
    }
  ]
}
```

## Backward Compatibility[​](#backward-compatibility "Direct link to Backward Compatibility")

Custom storage types ending in `_s3` (e.g., `minio_s3`, `custom_s3`) still work, but `storage_type: "s3"` is preferred.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### AWS S3[​](#aws-s3 "Direct link to AWS S3")

| Problem                           | Solution                                                                                                                   |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Access denied / credential errors | Verify credentials: `aws s3 ls s3://your-bucket --profile your-profile`. Check IAM includes `s3:PutObject`. Verify region. |
| Slow uploads                      | Enable S3 Transfer Acceleration. Use S3 in the same region as your source database.                                        |

### S3-Compatible Storage[​](#s3-compatible-storage-1 "Direct link to S3-Compatible Storage")

| Problem           | Solution                                                                                            |
| ----------------- | --------------------------------------------------------------------------------------------------- |
| Connection errors | Verify endpoint URL. Check firewall rules. Ensure valid SSL certificates. Verify credential format. |


---

# Intermediate Storage

After extracting data from a source database, LakeXpress stages the results as Parquet files in intermediate storage before publishing to a target platform (such as Snowflake or Databricks). This staging step decouples extraction from publishing: you can extract once and publish to multiple targets, re-publish without re-extracting, or inspect the raw files before they reach the destination.

LakeXpress supports two categories of intermediate storage:

| Option               | Flag                  | Description                                            |
| -------------------- | --------------------- | ------------------------------------------------------ |
| **Local Filesystem** | `--output_dir`        | Stage files in a local directory                       |
| **Cloud Storage**    | `--target_storage_id` | Stage files in S3, GCS, Azure Blob Storage, or OneLake |

`--output_dir` and `--target_storage_id` are mutually exclusive — choose one per pipeline run.

note

Cloud storage (S3, GCS, Azure, OneLake) is a staging location, not a publishing target. Publishing targets such as Snowflake, Databricks, and BigQuery are configured separately with `--publish_target`. See the [Snowflake Publishing Guide](/0.3/integrations/snowflake-publishing.md) for an example of how publishing follows storage.

## Sub-Path Option[​](#sub-path-option "Direct link to Sub-Path Option")

All backends support `--sub_path` to insert an intermediate directory level:

```
./LakeXpress --target_storage_id s3_01 --sub_path staging/daily/2025-01-15 ...
```

Produces:

```
s3://bucket/base_path/staging/daily/2025-01-15/schema_name/table_name/*.parquet
```

Use cases: date partitioning (`staging/2025/01/15`), environment separation (`dev/exports` vs `prod/exports`), project organization (`project-a/datasets`).

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/0.3/guides/quickstart.md)
* [CLI Reference](/0.3/reference/cli-reference.md)
* [Database Configuration](/0.3/configuration/database-configuration.md)
* [Examples & Recipes](/0.3/guides/command-cookbook.md)


---

# Examples & Recipes

Tested commands and real-world examples for common LakeXpress scenarios. Linux syntax shown; on Windows PowerShell, replace `\` with `` ` `` and `./LakeXpress` with `.\LakeXpress.exe`.

Required sync arguments

The `sync`, `sync[export]`, and `sync[publish]` commands require `-a`, `--lxdb_auth_id`, and `--sync_id`. The `<SYNC_ID>` in examples below refers to the sync ID output by `config create`.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Database Lifecycle Commands](#database-lifecycle-commands)

* [Configuration Management](#configuration-management)

* [Local Exports](#local-exports)

  * [PostgreSQL Source](#postgresql-source)
  * [SQL Server Source](#sql-server-source)
  * [MySQL Source](#mysql-source)
  * [MariaDB Source](#mariadb-source)
  * [Teradata Source](#teradata-source)

* [AWS S3 Exports](#aws-s3-exports)

* [Azure Storage Exports](#azure-storage-exports)

* [Google Cloud Storage Exports](#google-cloud-storage-exports)

* [OneLake Exports](#onelake-exports)

* [Snowflake Publishing](#snowflake-publishing)

  * [External Tables](#snowflake-external-tables)
  * [Internal Tables](#snowflake-internal-tables)

* [Databricks Publishing](#databricks-publishing)

* [AWS Glue Publishing](#aws-glue-publishing)

* [Microsoft Fabric Publishing](#microsoft-fabric-publishing)

* [Table Filtering](#table-filtering)

* [Incremental Sync](#incremental-sync)

* [Resume Failed Exports](#resume-failed-exports)

***

## Database Lifecycle Commands[​](#database-lifecycle-commands "Direct link to Database Lifecycle Commands")

```
# Initialize the LakeXpress DB schema
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_ms

# Drop the LakeXpress DB schema (requires --confirm)
./LakeXpress lxdb drop -a credentials.json --lxdb_auth_id lxdb_ms --confirm

# Clear all data, keep the schema
./LakeXpress lxdb truncate -a credentials.json --lxdb_auth_id lxdb_ms

# Show locked tables (incremental syncs only)
./LakeXpress lxdb locks -a credentials.json --lxdb_auth_id lxdb_ms

# Release stale or stuck locks
./LakeXpress lxdb release-locks -a credentials.json --lxdb_auth_id lxdb_ms --confirm
```

### Different LakeXpress DB Types[​](#different-lakexpress-db-types "Direct link to Different LakeXpress DB Types")

```
# SQL Server as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_ms_02

# PostgreSQL as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_pg

# DuckDB as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_duckdb

# MySQL as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id mysql_log_01

# SQLite as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id sqlite_01
```

***

## Configuration Management[​](#configuration-management "Direct link to Configuration Management")

```
# Create a new sync configuration
./LakeXpress config create [OPTIONS]

# List all sync configurations
./LakeXpress config list -a credentials.json --lxdb_auth_id lxdb_ms

# Delete a sync configuration
./LakeXpress config delete -a credentials.json --lxdb_auth_id lxdb_ms --sync_id <SYNC_ID>

# Run from a legacy YAML config file
./LakeXpress run -c config_20251222.yml
```

***

## Local Exports[​](#local-exports "Direct link to Local Exports")

### PostgreSQL Source[​](#postgresql-source "Direct link to PostgreSQL Source")

**PostgreSQL to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

### SQL Server Source[​](#sql-server-source "Direct link to SQL Server Source")

**SQL Server to Local (PostgreSQL LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (SQLite LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id sqlite_01 \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id sqlite_01 \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (DuckDB LakeXpress DB)**

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_duckdb

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_duckdb \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_duckdb \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (MySQL LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id mysql_log_01 \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id mysql_log_01 \
    --sync_id <SYNC_ID>
```

### MySQL Source[​](#mysql-source "Direct link to MySQL Source")

**MySQL to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id mysql_01 \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

### MariaDB Source[​](#mariadb-source "Direct link to MariaDB Source")

**MariaDB to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id maria_01 \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

### Teradata Source[​](#teradata-source "Direct link to Teradata Source")

**Teradata to Local with Timepartition**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg_01 \
    --source_db_auth_id source_teradata \
    --source_db_name DW \
    --source_schema_name claims \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4" \
    --fastbcp_table_config "PRESCRIPTION_DRUG_EVENTS:Timepartition:(SRVC_DT,year,month):4" \
    --n_jobs 4 \
    --output_dir /tmp/claims/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg_01 \
    --sync_id <SYNC_ID>
```

***

## AWS S3 Exports[​](#aws-s3-exports "Direct link to AWS S3 Exports")

**PostgreSQL to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**PostgreSQL to S3 (OVH)**

Uses OVH S3-compatible storage with custom endpoint via AWS profile:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --sync_id <SYNC_ID>
```

**credentials.json:**

```
{
  "s3_02": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-ovh-bucket/exports",
      "profile": "ovh"
    }
  }
}
```

**\~/.aws/config:**

```
[profile ovh]
endpoint_url = https://s3.gra.io.cloud.ovh.net
region = gra
```

**\~/.aws/credentials:**

```
[ovh]
aws_access_key_id = YOUR_OVH_ACCESS_KEY
aws_secret_access_key = YOUR_OVH_SECRET_KEY
```

**PostgreSQL to S3 with Sub-Path and Table Filter**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --sub_path staging/data \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MySQL to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id mysql_01 \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --sub_path mysql_export \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MariaDB to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id maria_01 \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --sub_path mariadb_export \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## Azure Storage Exports[​](#azure-storage-exports "Direct link to Azure Storage Exports")

**PostgreSQL to Azure**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id azure_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server to Azure**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg \
    --source_db_auth_id ds_11_ms \
    --source_db_name adventureworks2019 \
    --source_schema_name sales \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id azure_01 \
    --sub_path adventureworks/sales \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_pg \
    --sync_id <SYNC_ID>
```

**credentials.json (Azure with Service Principal):**

```
{
  "azure_01": {
    "ds_type": "azure",
    "auth_mode": "service_principal",
    "info": {
      "directory": "abfss://datalake.dfs.core.windows.net/exports",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  },
  "lxdb_pg": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "ds_11_ms": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "localhost",
      "port": 1433,
      "database": "adventureworks2019"
    }
  }
}
```

***

## Google Cloud Storage Exports[​](#google-cloud-storage-exports "Direct link to Google Cloud Storage Exports")

**PostgreSQL to GCS**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id gcs_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## OneLake Exports[​](#onelake-exports "Direct link to OneLake Exports")

**PostgreSQL to OneLake**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id fabric_onelake \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## Snowflake Publishing[​](#snowflake-publishing "Direct link to Snowflake Publishing")

Export to S3, then create Snowflake tables.

### Snowflake External Tables[​](#snowflake-external-tables "Direct link to Snowflake External Tables")

Data stays in S3; queryable via Snowflake.

**External Tables with Views (Default)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path snowflake_ext \
    --include "nation%" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**External Tables without Views**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path snowflake_ext \
    --include "nation%" \
    --publish_target snowflake_pat \
    --no_views

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

### Snowflake Internal Tables[​](#snowflake-internal-tables "Direct link to Snowflake Internal Tables")

Data loaded into Snowflake storage.

**Basic Internal Tables**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path snowflake_int \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**Internal Tables with Custom Naming**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path production \
    --publish_method internal \
    --publish_schema_pattern "INT_{subpath}_{date}" \
    --publish_table_pattern "{schema}_{table}" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**Internal Tables with Primary Key Constraints**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path production \
    --publish_method internal \
    --publish_schema_pattern "INT_{subpath}_{date}" \
    --publish_table_pattern "{schema}_{table}" \
    --publish_target snowflake_pat \
    --snowflake_pk_constraints

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server to Snowflake Internal**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --publish_method internal \
    --publish_schema_pattern "adventureworksdw" \
    --publish_table_pattern "{table}" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MySQL to Snowflake (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id mysql_01 \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path mysql_data \
    --include "nation%" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MySQL to Snowflake (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id mysql_01 \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path mysql_data \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MariaDB to Snowflake (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id maria_01 \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path mariadb_data \
    --include "nation%" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**MariaDB to Snowflake (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id maria_01 \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_01 \
    --generate_metadata \
    --sub_path mariadb_data \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_pat

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## Databricks Publishing[​](#databricks-publishing "Direct link to Databricks Publishing")

Export to S3, then create Databricks Unity Catalog tables.

**PostgreSQL to Databricks (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_02 \
    --publish_target databricks_01 \
    --publish_method external

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**PostgreSQL to Databricks (Internal/Managed Delta Tables)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_02 \
    --publish_target databricks_01 \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server to Databricks (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_02 \
    --publish_target databricks_01 \
    --publish_schema_pattern "adventureworksdw" \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## AWS Glue Publishing[​](#aws-glue-publishing "Direct link to AWS Glue Publishing")

Export to S3, then register tables in AWS Glue Data Catalog.

**PostgreSQL to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_01 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_01 \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**PostgreSQL to AWS Glue with Custom FastBCP Config**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_01 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:TO_CHAR(l_shipdate, 'YYYY'):8;orders:Ctid::4" \
    --n_jobs 2 \
    --publish_target glue_01 \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_09_ms \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_01 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_01 \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server to AWS Glue with Custom FastBCP Config**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_09_ms \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_01 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Physloc::4" \
    --n_jobs 2 \
    --publish_target glue_01 \
    --publish_database_name "lakexpress_tpch" \
    --publish_table_pattern "{schema}_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**SQL Server AdventureWorks to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_10_ms \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id aws_s3_02 \
    --publish_target glue_01 \
    --publish_schema_pattern "adventureworksdw"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## Microsoft Fabric Publishing[​](#microsoft-fabric-publishing "Direct link to Microsoft Fabric Publishing")

Export to OneLake, then create Fabric Lakehouse tables.

**PostgreSQL to Fabric (Internal/Delta Tables)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --target_storage_id fabric_onelake \
    --publish_target fabric_01 \
    --publish_method internal \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**PostgreSQL to Fabric (External/SQL Views)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --target_storage_id fabric_onelake \
    --publish_target fabric_01 \
    --publish_method external \
    --publish_table_pattern "VW_{schema}_{table}" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

***

## Table Filtering[​](#table-filtering "Direct link to Table Filtering")

### Include Specific Table Patterns[​](#include-specific-table-patterns "Direct link to Include Specific Table Patterns")

Export only fact and dimension tables:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --include "fact_%, dim_%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --output_dir ./exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Includes `fact_sales`, `fact_orders`, `dim_customer`, `dim_product`, etc.

### Exclude Temporary and Test Tables[​](#exclude-temporary-and-test-tables "Direct link to Exclude Temporary and Test Tables")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --exclude "temp_%, test_%, staging_%" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --output_dir ./exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

### Combine Include and Exclude[​](#combine-include-and-exclude "Direct link to Combine Include and Exclude")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name analytics \
    --include "report_%, dashboard_%" \
    --exclude "%_test, %_backup" \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --output_dir ./exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Includes `report_sales`, `dashboard_kpi`. Excludes `report_sales_test`, `dashboard_kpi_backup`.

***

## Incremental Sync[​](#incremental-sync "Direct link to Incremental Sync")

Watermark-based delta exports.

**Basic Incremental Sync**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --source_db_auth_id ds_04_pg \
    --source_db_name tpch \
    --source_schema_name tpch_1_incremental \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --target_storage_id aws_s3_01 \
    --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
    --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
    --generate_metadata

# First sync exports all data; subsequent syncs export only new/changed data
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms_02 \
    --sync_id <SYNC_ID>
```

**Daily Order Updates**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name ecommerce \
    --source_schema_name public \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --target_storage_id s3_01 \
    --incremental_table "public.orders:created_at:datetime" \
    --incremental_table "public.order_items:created_at:datetime" \
    --publish_target snowflake_prod \
    --n_jobs 4

# Run daily via cron or scheduler
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

First run exports all rows. Subsequent runs export only new orders; other tables are fully exported each time.

**Event Log Ingestion**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name events \
    --source_schema_name events \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --target_storage_id aws_s3_01 \
    --incremental_table "events.pageviews:event_time:timestamp" \
    --incremental_table "events.clicks:event_time:timestamp" \
    --incremental_table "events.conversions:event_time:timestamp" \
    --incremental_safety_lag 600 \
    --sub_path production/events \
    --n_jobs 8 \
    --fastbcp_p 4

# Run every 10 minutes
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

**Debug Mode**

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id <LXDB_AUTH_ID> \
    --sync_id <SYNC_ID> \
    --log_lev DEBUG
```

***

## Resume Failed Exports[​](#resume-failed-exports "Direct link to Resume Failed Exports")

When a sync starts, note the run ID from output:

```
2025-10-31 10:15:23 | INFO | Starting sync run: 20251031-2f73b4d0-8647-11ef-8089-c403a82a4577
```

Resume with:

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id <LXDB_AUTH_ID> \
    --sync_id <SYNC_ID> \
    --run_id 20251031-2f73b4d0-8647-11ef-8089-c403a82a4577 \
    --resume
```

This skips completed tables and retries failed ones.

### Query Failed Tables[​](#query-failed-tables "Direct link to Query Failed Tables")

```
SELECT
    source_schema,
    source_table,
    status,
    error_message,
    started_at,
    finished_at
FROM jobs
WHERE run_id = '20251031-2f73b4d0-8647-11ef-8089-c403a82a4577'
  AND status = 'failed'
ORDER BY source_schema, source_table;
```

***

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/0.3/guides/quickstart.md)
* [CLI Reference](/0.3/reference/cli-reference.md)
* [Incremental Sync](/0.3/advanced/incremental-sync.md)
* [![](/img/logos/snowflake.svg)Snowflake Publishing](/0.3/integrations/snowflake-publishing.md)
* [![](/img/logos/databricks.svg)Databricks Publishing](/0.3/integrations/databricks-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue Publishing](/0.3/integrations/glue-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Publishing](/0.3/integrations/fabric-publishing.md)


---

# Quick start guide

Two steps: create a config, then run a sync.

## Preparation[​](#preparation "Direct link to Preparation")

### 1. Install LakeXpress[​](#1-install-lakexpress "Direct link to 1. Install LakeXpress")

Download the LakeXpress archive for your platform and unzip it. FastBCP is bundled in the archive. See the [Installation guide](/0.3/installation.md) for details.

### 2. Prepare the source database[​](#2-prepare-the-source-database "Direct link to 2. Prepare the source database")

LakeXpress needs read-only access to export tables from your source database. Create a dedicated user with SELECT privileges on the schemas you want to export.

**PostgreSQL example:**

```
CREATE USER lakexpress_reader WITH PASSWORD 'your_password';
GRANT USAGE ON SCHEMA public TO lakexpress_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO lakexpress_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT ON TABLES TO lakexpress_reader;
```

For other databases, see the [Database Configuration](/0.3/configuration/database-configuration.md) page (Oracle, SQL Server, MySQL, MariaDB, SAP HANA).

### 3. Prepare the LakeXpress DB[​](#3-prepare-the-lakexpress-db "Direct link to 3. Prepare the LakeXpress DB")

LakeXpress needs its own database to store configuration, track syncs, and log runs. It creates the schema automatically on first run -- you just need an empty database with a user that has full privileges.

**PostgreSQL example:**

```
CREATE DATABASE lakexpress_log;
CREATE USER lakexpress_admin WITH PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE lakexpress_log TO lakexpress_admin;
```

Quick testing

For quick testing, SQLite or DuckDB need no server -- just point to a file path in your credentials file.

For SQL Server or MySQL alternatives, see [Database Configuration](/0.3/configuration/database-configuration.md).

### 4. Set up storage[​](#4-set-up-storage "Direct link to 4. Set up storage")

For the simplest setup, create a local directory to receive the exported Parquet files:

```
mkdir ./exports
```

For cloud storage (S3, GCS, Azure Blob), see [Storage Overview](/0.3/configuration/storage/storage-overview.md).

## Create a credentials file[​](#create-a-credentials-file "Direct link to Create a credentials file")

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-data-lake/exports",
      "profile": "your-aws-profile"
    }
  }
}
```

Save as `credentials.json` in a secure location.

Environment variables

Use `$env{VAR_NAME}` in any string value. An error is raised if the variable is not set. Plain-text values also work.

```
# Linux
export LX_PG_USER="postgres"
export LX_PG_PASSWORD="your_password"

# Windows (cmd)
set LX_PG_USER=postgres
set LX_PG_PASSWORD=your_password

# Windows (PowerShell)
$env:LX_PG_USER = "postgres"
$env:LX_PG_PASSWORD = "your_password"
```

For other databases (Oracle, SQL Server, MySQL) and storage backends (GCS, Azure), see [Database Configuration](/0.3/configuration/database-configuration.md) and [Intermediate Storage](/0.3/configuration/storage/storage-overview.md).

## Initialize the LakeXpress DB (optional)[​](#initialize-the-lakexpress-db-optional "Direct link to Initialize the LakeXpress DB (optional)")

The LakeXpress DB tracks syncs, runs, and table exports. LakeXpress creates the schema automatically on first sync, so this step is optional.

Use `lxdb init` to verify connectivity or pre-create the schema for audit purposes.

### Windows (PowerShell)[​](#windows-powershell "Direct link to Windows (PowerShell)")

```
.\LakeXpress.exe lxdb init `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres
```

### Linux[​](#linux "Direct link to Linux")

```
./LakeXpress lxdb init \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

## Create a sync configuration[​](#create-a-sync-configuration "Direct link to Create a sync configuration")

The configuration is stored in the LakeXpress DB and reused for every sync.

### Export to local filesystem[​](#export-to-local-filesystem "Direct link to Export to local filesystem")

#### Windows (PowerShell)[​](#windows-powershell-1 "Direct link to Windows (PowerShell)")

```
.\LakeXpress.exe config create `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --source_db_auth_id source_postgres `
  --source_db_name public `
  --source_schema_name public `
  --fastbcp_dir_path .\FastBCP_win-x64\latest\ `
  --output_dir .\exports `
  --n_jobs 4 `
  --fastbcp_p 2
```

#### Linux[​](#linux-1 "Direct link to Linux")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name public \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --output_dir ./exports \
  --n_jobs 4 \
  --fastbcp_p 2
```

Exports all tables from `public` to `./exports/public/table_name/`, 4 tables in parallel with 2-way partitioning.

## Run the sync[​](#run-the-sync "Direct link to Run the sync")

info

The `sync` command requires the same `-a` and `--lxdb_auth_id` used in `config create`, plus `--sync_id` (printed by `config create`).

### Windows (PowerShell)[​](#windows-powershell-2 "Direct link to Windows (PowerShell)")

```
.\LakeXpress.exe sync `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID>
```

### Linux[​](#linux-2 "Direct link to Linux")

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Loads the config from the LakeXpress DB, exports tables, and shows real-time progress.

***

## More examples[​](#more-examples "Direct link to More examples")

### Export to cloud storage[​](#export-to-cloud-storage "Direct link to Export to cloud storage")

Export to AWS S3 with CDM metadata:

#### Windows (PowerShell)[​](#windows-powershell-3 "Direct link to Windows (PowerShell)")

```
.\LakeXpress.exe config create `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --source_db_auth_id source_postgres `
  --source_db_name tpch `
  --source_schema_name public `
  --fastbcp_dir_path .\FastBCP_win-x64\latest\ `
  --target_storage_id s3_01 `
  --n_jobs 4 `
  --fastbcp_p 2 `
  --generate_metadata

.\LakeXpress.exe sync `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID>
```

#### Linux[​](#linux-3 "Direct link to Linux")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name tpch \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --n_jobs 4 \
  --fastbcp_p 2 \
  --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Exports to S3 and generates CDM metadata files.

### Filter tables with patterns[​](#filter-tables-with-patterns "Direct link to Filter tables with patterns")

Use include/exclude patterns to select specific tables:

#### Windows (PowerShell)[​](#windows-powershell-4 "Direct link to Windows (PowerShell)")

```
.\LakeXpress.exe config create `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --source_db_auth_id source_postgres `
  --source_db_name public `
  --source_schema_name public `
  --include "orders%, customer%, product%" `
  --exclude "temp%, test%" `
  --fastbcp_dir_path .\FastBCP_win-x64\latest\ `
  --output_dir .\exports `
  --n_jobs 4

.\LakeXpress.exe sync `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID>
```

#### Linux[​](#linux-4 "Direct link to Linux")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name public \
  --source_schema_name public \
  --include "orders%, customer%, product%" \
  --exclude "temp%, test%" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --output_dir ./exports \
  --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Includes tables matching `orders%`, `customer%`, or `product%`; excludes those matching `temp%` or `test%`.

### Incremental sync[​](#incremental-sync "Direct link to Incremental sync")

Use a watermark column so subsequent syncs only export new rows:

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id source_pg \
  --source_db_name tpch \
  --source_schema_name tpch_1_incremental \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
  --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
  --generate_metadata

# First sync: exports everything and records high watermarks
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --sync_id <SYNC_ID>

# Later syncs: only exports rows past the watermark
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --sync_id <SYNC_ID>
```

Tables not configured as incremental are fully exported each sync. See the [Incremental Sync](/0.3/advanced/incremental-sync.md) guide for details.

### Resume failed syncs[​](#resume-failed-syncs "Direct link to Resume failed syncs")

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID> \
  --run_id 20251208-f7g8h9i0-j1k2-l3m4 \
  --resume
```

Skips completed tables and retries only the failed ones.

### Snowflake publishing[​](#snowflake-publishing "Direct link to Snowflake publishing")

Export to S3 and create Snowflake external tables in one step:

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name public \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --publish_target snowflake_prod \
  --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Query the data in Snowflake:

```
SELECT * FROM PUBLIC.V_CUSTOMER LIMIT 10;
```

For internal tables with primary key constraints, add `--publish_method internal --snowflake_pk_constraints`. See the [Snowflake Publishing Guide](/0.3/integrations/snowflake-publishing.md).

## Reference[​](#reference "Direct link to Reference")

### List configurations[​](#list-configurations "Direct link to List configurations")

```
./LakeXpress config list \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

### Check sync status[​](#check-sync-status "Direct link to Check sync status")

```
./LakeXpress status -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <your-sync-id>
```

### Manage the LakeXpress DB[​](#manage-the-lakexpress-db "Direct link to Manage the LakeXpress DB")

```
# Initialize the schema
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_postgres

# Clear run history (keeps schema)
./LakeXpress lxdb truncate -a credentials.json --lxdb_auth_id lxdb_postgres

# Drop the schema
./LakeXpress lxdb drop -a credentials.json --lxdb_auth_id lxdb_postgres --confirm
```

## Next steps[​](#next-steps "Direct link to Next steps")

* [CLI reference](/0.3/reference/cli-reference.md) - all available options
* [Incremental sync](/0.3/advanced/incremental-sync.md) - continuous updates
* [Intermediate Storage](/0.3/configuration/storage/storage-overview.md) - S3, GCS, Azure, local
* [Database configuration](/0.3/configuration/database-configuration.md) - all supported databases
* [Examples & Recipes](/0.3/guides/command-cookbook.md) - real-world scenarios


---

# Installation

## Instructions[​](#instructions "Direct link to Instructions")

* Windows
* Linux

To start using LakeXpress, first **[request a trial](https://arpe.io/get-trial?product=LakeXpress)** to download the latest version.

Once downloaded, unzip LakeXpress to a directory of your choice.

```
Expand-Archive -Path .\LakeXpress.zip -DestinationPath .\LakeXpress
```

You can verify the installation by checking the version:

```
.\LakeXpress.exe --version
```

To start using LakeXpress, first **[request a trial](https://arpe.io/get-trial?product=LakeXpress)** to download the latest version.

Once downloaded, unzip LakeXpress to a directory of your choice.

```
unzip LakeXpress.zip -d LakeXpress
```

On **Linux**, you need to give execution rights to the LakeXpress binary:

```
chmod +x LakeXpress
```

<br />

You can verify the installation by checking the version:

```
./LakeXpress --version
```

<br />

**You are now ready to run LakeXpress.**

## What's Included[​](#whats-included "Direct link to What's Included")

The trial edition includes the following directory structure:

* Windows
* Linux

```
LakeXpress/
├── LakeXpress.exe           - Main executable
├── engine/
│   └── FastBCP.exe          - Bundled FastBCP engine (auto-discovered)
└── credentials.json         - Sample authentication file
```

```
LakeXpress/
├── LakeXpress               - Main executable
├── engine/
│   └── FastBCP              - Bundled FastBCP engine (auto-discovered)
└── credentials.json         - Sample authentication file
```

Bundled FastBCP Engine

Trial editions include a bundled FastBCP engine in the `engine/` directory that is auto-discovered — no need to specify `--fastbcp_dir_path`.

## License Information[​](#license-information "Direct link to License Information")

License Information

After the trial period you will need a valid license. LakeXpress resolves the license in the following order:

1. **`LAKEXPRESS_LICENSE` environment variable** — set this for CI/CD or containerized deployments

2. **`--license TEXT`** CLI argument — pass the license content directly

3. **`--license_file PATH`** CLI argument — point to a license file

4. **Default file locations:**

   * `~/.lakexpress/license.lic` (Linux/Mac)
   * `%APPDATA%/LakeXpress/license.lic` (Windows)
   * `./license.lic` (current directory)

5. **Trial mode** — automatic 30-day trial if no license is found

See the [CLI Reference](/0.3/reference/cli-reference.md#license-options) for details.


---

# BigQuery Publishing

LakeXpress can **automatically create BigQuery tables** from Parquet files stored in GCS.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication](#authentication)
* [Configuration Options](#configuration-options)
* [Table Types](#table-types)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Google Cloud Permissions[​](#1-google-cloud-permissions "Direct link to 1. Google Cloud Permissions")

Required IAM roles:

* **BigQuery Data Editor** (`roles/bigquery.dataEditor`) - Create datasets and tables
* **BigQuery Job User** (`roles/bigquery.jobUser`) - Run load jobs (native tables)
* **Storage Object Viewer** (`roles/storage.objectViewer`) - Read GCS files (external tables)

Or a custom role with these permissions:

```
bigquery.datasets.create
bigquery.datasets.get
bigquery.tables.create
bigquery.tables.get
bigquery.tables.delete
bigquery.tables.updateData
bigquery.jobs.create
storage.objects.get
storage.objects.list
```

### 2. GCS Storage Setup[​](#2-gcs-storage-setup "Direct link to 2. GCS Storage Setup")

Your `credentials.json` needs both GCS and BigQuery entries:

```
{
    "gcs_datalake": {
        "ds_type": "gcs",
        "auth_mode": "profile",
        "info": {
            "directory": "gs://my-datalake-bucket/lakexpress/",
            "profile": "/path/to/service-account.json"
        }
    },
    "bigquery_prod": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-gcp-project",
            "location": "US",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

The same service account JSON file can be used for both GCS and BigQuery.

## Authentication[​](#authentication "Direct link to Authentication")

LakeXpress uses service account authentication for BigQuery.

```
{
    "bigquery_prod": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-gcp-project",
            "location": "US",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

| Field        | Required | Description                                   |
| ------------ | -------- | --------------------------------------------- |
| `ds_type`    | Yes      | Must be `"bigquery"`                          |
| `auth_mode`  | Yes      | Must be `"profile"`                           |
| `project_id` | Yes      | GCP project ID where datasets will be created |
| `location`   | No       | BigQuery dataset location (default: `"US"`)   |
| `profile`    | Yes      | Path to service account JSON key file         |

Common locations: `US`, `EU`, `us-central1`, `europe-west1`.

### Creating a Service Account[​](#creating-a-service-account "Direct link to Creating a Service Account")

1. Go to Google Cloud Console > IAM & Admin > Service Accounts
2. Create a service account with BigQuery and Storage permissions
3. Create and download a JSON key
4. Reference the key file path in your credentials

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                  | Default    |
| ---------------------------------- | -------------------------------------------- | ---------- |
| `--publish_target ID`              | Credential ID for BigQuery target (required) | -          |
| `--publish_schema_pattern PATTERN` | BigQuery dataset naming pattern              | `{schema}` |
| `--publish_table_pattern PATTERN`  | Table naming pattern                         | `{table}`  |
| `--publish_method TYPE`            | Table type: `external` or `internal`         | `external` |
| `--n_jobs N`                       | Parallel workers for table creation          | `1`        |

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables (Default)[​](#external-tables-default "Direct link to External Tables (Default)")

Queries data directly from GCS Parquet files without loading into BigQuery storage.

```
lakexpress ... --publish_target bigquery_prod --publish_method external
```

* Fast publishing, no data copying
* No BigQuery storage costs
* Always reflects latest GCS data
* Best for large datasets with infrequent queries
* Query speed depends on GCS read performance
* No clustering or partitioning support

### Native Tables[​](#native-tables "Direct link to Native Tables")

Loads data from GCS into BigQuery's columnar storage.

```
lakexpress ... --publish_target bigquery_prod --publish_method internal
```

* Faster queries via optimized columnar storage
* Full BigQuery features (clustering, partitioning, materialized views)
* Best for frequently queried data
* Slower publishing due to data loading
* Incurs BigQuery storage costs
* Data is a point-in-time snapshot

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Dataset and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                           | Example Output    |
| ------------- | ------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                    | `tpch_1`          |
| `{table}`     | Source table name                     | `customer`        |
| `{database}`  | Source database name                  | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)               | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  | `20251210_143022` |
| `{uuid}`      | UUID4 identifier (consistent per run) | `a1b2c3d4`        |
| `{subpath}`   | CLI `--sub_path` parameter            | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### Prefixed Datasets[​](#prefixed-datasets "Direct link to Prefixed Datasets")

```
lakexpress \
  --publish_schema_pattern "lx_{schema}" \
  --publish_table_pattern "{table}" \
  --publish_target bigquery_prod \
  ...

# Results:
# Dataset: lx_tpch_1
# Tables: customer, orders, lineitem
```

#### Date-Partitioned Datasets[​](#date-partitioned-datasets "Direct link to Date-Partitioned Datasets")

```
lakexpress \
  --publish_schema_pattern "{schema}_{date}" \
  --publish_table_pattern "{table}" \
  --publish_target bigquery_prod \
  ...

# Results:
# Dataset: tpch_1_20251210
# Tables: customer, orders, lineitem
```

#### Consolidated Multi-Schema[​](#consolidated-multi-schema "Direct link to Consolidated Multi-Schema")

```
lakexpress \
  --source_schema_name schema1,schema2 \
  --publish_schema_pattern "consolidated" \
  --publish_table_pattern "{schema}_{table}" \
  --publish_target bigquery_prod \
  ...

# Results:
# Dataset: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: External Tables (Default)[​](#example-1-external-tables-default "Direct link to Example 1: External Tables (Default)")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id gcs_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target bigquery_prod
```

Creates dataset `public` with external tables pointing to GCS Parquet files.

### Example 2: Native Tables with Parallel Execution[​](#example-2-native-tables-with-parallel-execution "Direct link to Example 2: Native Tables with Parallel Execution")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id gcs_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target bigquery_prod \
    --publish_method internal \
    --publish_schema_pattern "lx_{schema}" \
    --n_jobs 4
```

Creates dataset `lx_tpch_1` with native tables loaded from GCS.

### Example 3: Daily Snapshots[​](#example-3-daily-snapshots "Direct link to Example 3: Daily Snapshots")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name sales \
    --target_storage_id gcs_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target bigquery_prod \
    --publish_schema_pattern "sales_{date}" \
    --sub_path "daily/$(date +%Y%m%d)"
```

Creates dataset `sales_20251210`, data stored in `gs://bucket/lakexpress/daily/20251210/`.

### Example 4: EU Data Residency[​](#example-4-eu-data-residency "Direct link to Example 4: EU Data Residency")

```
{
    "bigquery_eu": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-eu-project",
            "location": "EU",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name gdpr_data \
    --target_storage_id gcs_eu \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target bigquery_eu
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/bigquery.svg)BigQuery Reference](/0.3/integrations/bigquery-reference.md)Data type mappings, CLI arguments, querying, troubleshooting
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# BigQuery Reference

Reference material for [BigQuery Publishing](/0.3/integrations/bigquery-publishing.md) -- data type mappings, CLI arguments, querying, and troubleshooting.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source database types are mapped to BigQuery types automatically.

### PostgreSQL to BigQuery[​](#postgresql-to-bigquery "Direct link to PostgreSQL to BigQuery")

| PostgreSQL Type      | BigQuery Type |
| -------------------- | ------------- |
| `INTEGER`, `INT4`    | `INT64`       |
| `BIGINT`, `INT8`     | `INT64`       |
| `SMALLINT`, `INT2`   | `INT64`       |
| `NUMERIC(p,s)`       | `NUMERIC`     |
| `REAL`, `FLOAT4`     | `FLOAT64`     |
| `DOUBLE PRECISION`   | `FLOAT64`     |
| `VARCHAR(n)`, `TEXT` | `STRING`      |
| `DATE`               | `DATE`        |
| `TIMESTAMP`          | `TIMESTAMP`   |
| `TIMESTAMPTZ`        | `TIMESTAMP`   |
| `TIME`               | `TIME`        |
| `BOOLEAN`            | `BOOL`        |
| `BYTEA`              | `BYTES`       |
| `JSON`, `JSONB`      | `JSON`        |
| `UUID`               | `STRING`      |

### SQL Server to BigQuery[​](#sql-server-to-bigquery "Direct link to SQL Server to BigQuery")

| SQL Server Type             | BigQuery Type |
| --------------------------- | ------------- |
| `INT`                       | `INT64`       |
| `BIGINT`                    | `INT64`       |
| `SMALLINT`                  | `INT64`       |
| `TINYINT`                   | `INT64`       |
| `DECIMAL(p,s)`              | `NUMERIC`     |
| `MONEY`                     | `NUMERIC`     |
| `FLOAT`                     | `FLOAT64`     |
| `REAL`                      | `FLOAT64`     |
| `VARCHAR(n)`, `NVARCHAR(n)` | `STRING`      |
| `DATE`                      | `DATE`        |
| `DATETIME`, `DATETIME2`     | `DATETIME`    |
| `DATETIMEOFFSET`            | `TIMESTAMP`   |
| `TIME`                      | `TIME`        |
| `BIT`                       | `BOOL`        |
| `VARBINARY`                 | `BYTES`       |
| `UNIQUEIDENTIFIER`          | `STRING`      |

### Oracle to BigQuery[​](#oracle-to-bigquery "Direct link to Oracle to BigQuery")

| Oracle Type             | BigQuery Type |
| ----------------------- | ------------- |
| `NUMBER`                | `NUMERIC`     |
| `NUMBER(p,s)`           | `NUMERIC`     |
| `FLOAT`                 | `FLOAT64`     |
| `VARCHAR2`, `NVARCHAR2` | `STRING`      |
| `CLOB`, `NCLOB`         | `STRING`      |
| `DATE`                  | `TIMESTAMP`   |
| `TIMESTAMP`             | `TIMESTAMP`   |
| `RAW`, `BLOB`           | `BYTES`       |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### BigQuery Publishing Arguments[​](#bigquery-publishing-arguments "Direct link to BigQuery Publishing Arguments")

| Option                             | Type    | Description                                          |
| ---------------------------------- | ------- | ---------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for BigQuery publishing (required)     |
| `--publish_schema_pattern PATTERN` | String  | Dynamic dataset naming pattern (default: `{schema}`) |
| `--publish_table_pattern PATTERN`  | String  | Dynamic table naming pattern (default: `{table}`)    |
| `--publish_method TYPE`            | String  | Table type: `external` (default) or `internal`       |
| `--n_jobs N`                       | Integer | Parallel workers for table creation (default: 1)     |

### Querying BigQuery Tables[​](#querying-bigquery-tables "Direct link to Querying BigQuery Tables")

**BigQuery Console:**

```
SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10;
```

**bq CLI:**

```
bq query --use_legacy_sql=false \
  'SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10'
```

**Python (google-cloud-bigquery):**

```
from google.cloud import bigquery

client = bigquery.Client()
query = "SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10"
df = client.query(query).to_dataframe()
print(df)
```

**pandas-gbq:**

```
import pandas_gbq

query = "SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10"
df = pandas_gbq.read_gbq(query, project_id="my-project")
print(df)
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

**"Permission denied" errors:**

* Verify the service account has BigQuery Data Editor role
* Check GCS bucket access from the service account
* Ensure project ID in credentials matches the target project

**"Dataset not found" errors:**

* Check that `location` matches your GCS bucket region
* Cross-region access between GCS and BigQuery may cause issues

**"Invalid table" errors for external tables:**

* Verify GCS path contains valid Parquet files
* Check schema mapping for your data types

### Verifying Setup[​](#verifying-setup "Direct link to Verifying Setup")

Test BigQuery connectivity:

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id gcs_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target bigquery_prod \
    --tables customer \
    --dry_run
```

Validate credentials:

```
from google.cloud import bigquery
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file(
    '/path/to/service-account.json'
)
client = bigquery.Client(credentials=credentials, project='my-project')
print(list(client.list_datasets()))
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/bigquery.svg)BigQuery Publishing](/0.3/integrations/bigquery-publishing.md)Setup and usage guide
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# Databricks Publishing

LakeXpress creates **Databricks Unity Catalog tables** from exported Parquet files. Supports external tables (data stays in S3) and managed Delta tables (data loaded into Databricks).

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication](#authentication)
* [Table Types](#table-types)
* [Configuration Options](#configuration-options)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Databricks Setup[​](#1-databricks-setup "Direct link to 1. Databricks Setup")

**Required Resources:**

* Databricks Workspace with Unity Catalog enabled
* SQL Warehouse (for DDL execution)
* Catalog with CREATE SCHEMA and CREATE TABLE permissions
* Storage credential for S3 access (external tables only)

**Required Permissions:**

* `CREATE SCHEMA` on the target catalog
* `CREATE TABLE` on the target schema
* `USE CATALOG` on the target catalog
* For external tables: access to the external location

### 2. Storage Credential Setup[​](#2-storage-credential-setup "Direct link to 2. Storage Credential Setup")

For external tables, configure a storage credential for S3 access:

```
-- In Databricks SQL
CREATE STORAGE CREDENTIAL my_s3_credential
WITH (
    AWS_IAM_ROLE = 'arn:aws:iam::123456789012:role/DatabricksS3AccessRole'
);

CREATE EXTERNAL LOCATION my_s3_location
URL 's3://my-datalake-bucket/lakexpress/'
WITH (STORAGE CREDENTIAL my_s3_credential);
```

### 3. Credentials Configuration[​](#3-credentials-configuration "Direct link to 3. Credentials Configuration")

Both S3 and Databricks credentials are needed in `credentials.json`:

```
{
    "aws_s3": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://your-bucket-name/path/to/exports",
            "profile": "your-aws-profile"
        }
    },
    "databricks": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}
```

## Authentication[​](#authentication "Direct link to Authentication")

Databricks uses Personal Access Tokens (PAT):

```
{
    "databricks": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}
```

**Configuration Fields:**

| Field          | Description                   | Required |
| -------------- | ----------------------------- | -------- |
| `host`         | Databricks workspace hostname | Yes      |
| `http_path`    | SQL Warehouse HTTP path       | Yes      |
| `access_token` | Personal Access Token (PAT)   | Yes      |
| `catalog`      | Target Unity Catalog          | Yes      |
| `schema`       | Default schema                | No       |

**To get the HTTP path:**

1. Go to Databricks SQL > SQL Warehouses
2. Select your warehouse > Connection details
3. Copy the HTTP path

**To create a PAT:**

1. Go to User Settings > Developer > Access tokens
2. Generate a new token

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables[​](#external-tables "Direct link to External Tables")

Reference Parquet data in S3. Databricks stores only metadata; queries read directly from S3.

* No data loading time or Databricks storage costs
* Data stays in place
* Available right after export

```
lakexpress -a credentials.json \
    --source_db_auth_id postgres_prod \
    --target_storage_id aws_s3_datalake \
    --publish_target databricks_prod \
    --databricks_table_type external  # Default
```

### Managed Tables (Delta)[​](#managed-tables-delta "Direct link to Managed Tables (Delta)")

Delta tables loaded into Databricks managed storage via COPY INTO. Enables ACID transactions, time travel, and Z-ordering.

* Faster queries with optimized storage and caching
* Full Delta Lake features
* Better for frequently accessed data

```
lakexpress -a credentials.json \
    --source_db_auth_id postgres_prod \
    --target_storage_id aws_s3_datalake \
    --publish_target databricks_prod \
    --databricks_table_type managed
```

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                    | Default     |
| ---------------------------------- | ---------------------------------------------- | ----------- |
| `--publish_target ID`              | Credential ID for Databricks target (required) | -           |
| `--publish_schema_pattern PATTERN` | Schema naming pattern                          | `{schema}`  |
| `--publish_table_pattern PATTERN`  | Table naming pattern                           | `{table}`   |
| `--databricks_table_type TYPE`     | `external` or `managed`                        | `external`  |
| `--databricks_catalog CATALOG`     | Override catalog from credentials              | (from auth) |
| `--n_jobs N`                       | Parallel workers for table creation            | `1`         |

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                          | Example Output    |
| ------------- | ------------------------------------ | ----------------- |
| `{schema}`    | Source schema name                   | `tpch_1`          |
| `{table}`     | Source table name                    | `customer`        |
| `{database}`  | Source database name                 | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)              | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS) | `20251210_143022` |
| `{uuid}`      | UUID4, consistent per run            | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` value               | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### Prefixed Schemas[​](#prefixed-schemas "Direct link to Prefixed Schemas")

```
lakexpress \
    --publish_schema_pattern "bronze_{schema}" \
    --publish_table_pattern "{table}" \
    --publish_target databricks_prod \
    ...

# Result:
# lakexpress_catalog.bronze_tpch_1.customer
# lakexpress_catalog.bronze_tpch_1.orders
```

#### Date-Based Schemas[​](#date-based-schemas "Direct link to Date-Based Schemas")

```
lakexpress \
    --publish_schema_pattern "{schema}_{date}" \
    --publish_table_pattern "{table}" \
    --publish_target databricks_prod \
    ...

# Result:
# lakexpress_catalog.tpch_1_20251210.customer
```

#### Medallion Architecture[​](#medallion-architecture "Direct link to Medallion Architecture")

```
# Bronze layer (raw data)
lakexpress \
    --publish_schema_pattern "bronze_{schema}" \
    --databricks_table_type external \
    --publish_target databricks_prod \
    ...

# Silver layer (curated data)
lakexpress \
    --publish_schema_pattern "silver_{schema}" \
    --databricks_table_type managed \
    --publish_target databricks_prod \
    ...
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: External Tables (Data in S3)[​](#example-1-external-tables-data-in-s3 "Direct link to Example 1: External Tables (Data in S3)")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target databricks_prod \
    --databricks_table_type external
```

Result: External Parquet tables in `lakexpress_catalog.public` reading from S3.

### Example 2: Managed Delta Tables[​](#example-2-managed-delta-tables "Direct link to Example 2: Managed Delta Tables")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target databricks_prod \
    --databricks_table_type managed \
    --n_jobs 4
```

Result: Delta tables in `lakexpress_catalog.tpch_1` loaded via COPY INTO.

### Example 3: Custom Catalog and Schema Pattern[​](#example-3-custom-catalog-and-schema-pattern "Direct link to Example 3: Custom Catalog and Schema Pattern")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name sales \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target databricks_prod \
    --databricks_catalog analytics_catalog \
    --publish_schema_pattern "raw_{schema}" \
    --databricks_table_type external
```

Result: Tables in `analytics_catalog.raw_sales` (`customer`, `orders`, etc.).

### Example 4: Parallel Table Creation[​](#example-4-parallel-table-creation "Direct link to Example 4: Parallel Table Creation")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target databricks_prod \
    --n_jobs 8
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/databricks.svg)Databricks Reference](/0.3/integrations/databricks-reference.md)Data type mappings, CLI arguments, querying, Delta features
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# Databricks Reference

Reference material for [Databricks Publishing](/0.3/integrations/databricks-publishing.md) -- data type mappings, CLI arguments, and querying.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types are mapped to Databricks types automatically.

### PostgreSQL to Databricks[​](#postgresql-to-databricks "Direct link to PostgreSQL to Databricks")

| PostgreSQL Type      | Databricks Type |
| -------------------- | --------------- |
| `INTEGER`, `INT4`    | `INT`           |
| `BIGINT`, `INT8`     | `BIGINT`        |
| `SMALLINT`, `INT2`   | `SMALLINT`      |
| `NUMERIC(p,s)`       | `DECIMAL(p,s)`  |
| `REAL`, `FLOAT4`     | `FLOAT`         |
| `DOUBLE PRECISION`   | `DOUBLE`        |
| `VARCHAR(n)`, `TEXT` | `STRING`        |
| `DATE`               | `DATE`          |
| `TIMESTAMP`          | `TIMESTAMP_NTZ` |
| `TIMESTAMPTZ`        | `TIMESTAMP`     |
| `BOOLEAN`            | `BOOLEAN`       |
| `BYTEA`              | `BINARY`        |

### SQL Server to Databricks[​](#sql-server-to-databricks "Direct link to SQL Server to Databricks")

| SQL Server Type             | Databricks Type |
| --------------------------- | --------------- |
| `INT`                       | `INT`           |
| `BIGINT`                    | `BIGINT`        |
| `SMALLINT`                  | `SMALLINT`      |
| `TINYINT`                   | `TINYINT`       |
| `DECIMAL(p,s)`              | `DECIMAL(p,s)`  |
| `FLOAT`                     | `DOUBLE`        |
| `REAL`                      | `FLOAT`         |
| `VARCHAR(n)`, `NVARCHAR(n)` | `STRING`        |
| `DATE`                      | `DATE`          |
| `DATETIME`, `DATETIME2`     | `TIMESTAMP_NTZ` |
| `DATETIMEOFFSET`            | `TIMESTAMP`     |
| `BIT`                       | `BOOLEAN`       |
| `VARBINARY`                 | `BINARY`        |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Databricks Publishing Arguments[​](#databricks-publishing-arguments "Direct link to Databricks Publishing Arguments")

| Option                             | Type    | Description                                        |
| ---------------------------------- | ------- | -------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for Databricks publishing (required) |
| `--publish_schema_pattern PATTERN` | String  | Schema naming pattern (default: `{schema}`)        |
| `--publish_table_pattern PATTERN`  | String  | Table naming pattern (default: `{table}`)          |
| `--databricks_table_type TYPE`     | Enum    | `external` or `managed` (default: `external`)      |
| `--databricks_catalog CATALOG`     | String  | Override catalog from credentials                  |
| `--n_jobs N`                       | Integer | Parallel workers for table creation (default: 1)   |

### Querying Tables[​](#querying-tables "Direct link to Querying Tables")

**Databricks SQL:**

```
SELECT * FROM lakexpress_catalog.tpch_1.customer LIMIT 10;
```

**Databricks Notebooks (Python):**

```
df = spark.table("lakexpress_catalog.tpch_1.customer")
df.display()
```

**Databricks Notebooks (SQL):**

```
%sql
SELECT * FROM lakexpress_catalog.tpch_1.customer LIMIT 10
```

### Delta Table Features (Managed Tables Only)[​](#delta-table-features-managed-tables-only "Direct link to Delta Table Features (Managed Tables Only)")

**Time Travel:**

```
SELECT * FROM lakexpress_catalog.tpch_1.customer VERSION AS OF 1;
```

**Table History:**

```
DESCRIBE HISTORY lakexpress_catalog.tpch_1.customer;
```

**Optimize:**

```
OPTIMIZE lakexpress_catalog.tpch_1.customer;
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/databricks.svg)Databricks Publishing](/0.3/integrations/databricks-publishing.md)Setup and usage guide
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# ducklake-publishing


---

# Microsoft Fabric Publishing

LakeXpress creates Microsoft Fabric Lakehouse tables from exported Parquet files. Query data in Fabric using SQL, Spark, or Power BI via managed Delta tables or SQL analytics views.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication Setup](#authentication-setup)
* [Table Types](#table-types)
* [Configuration Options](#configuration-options)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Microsoft Fabric Setup[​](#1-microsoft-fabric-setup "Direct link to 1. Microsoft Fabric Setup")

**Required Resources:**

* Microsoft Fabric workspace with a Lakehouse
* Service Principal with appropriate permissions
* OneLake storage access

**Required Permissions:**

* **Workspace**: Member or Contributor role
* **Lakehouse**: Read and Write access
* **OneLake**: Storage Blob Data Contributor

### 2. Azure AD App Registration[​](#2-azure-ad-app-registration "Direct link to 2. Azure AD App Registration")

Create a Service Principal:

1. **Azure Portal** > **Azure Active Directory** > **App registrations**
2. Click **New registration**
3. Name the app (e.g., "LakeXpress-Fabric")
4. Select **Accounts in this organizational directory only**
5. Click **Register**
6. Note the **Application (client) ID** and **Directory (tenant) ID**
7. Go to **Certificates & secrets** > **New client secret**
8. Create a secret and save the **Value** (shown only once)

### 3. Grant Fabric Permissions[​](#3-grant-fabric-permissions "Direct link to 3. Grant Fabric Permissions")

Add the Service Principal to your workspace:

1. Open your **Fabric workspace**
2. Click **Manage access**
3. Click **Add people or groups**
4. Search for your Service Principal
5. Assign **Member** or **Contributor** role
6. Click **Add**

### 4. Find Fabric Configuration Values[​](#4-find-fabric-configuration-values "Direct link to 4. Find Fabric Configuration Values")

**Workspace ID and Lakehouse ID:**

1. Open your Lakehouse in the Fabric portal
2. Extract from the URL:
   <!-- -->
   ```
   https://app.fabric.microsoft.com/groups/{workspace_id}/lakehouses/{lakehouse_id}
   ```
3. Copy `workspace_id` (GUID after `/groups/`)
4. Copy `lakehouse_id` (GUID after `/lakehouses/`)

**Lakehouse Name:**

The display name shown in the Fabric portal.

**SQL Analytics Endpoint:**

1. In your Lakehouse, click **SQL analytics endpoint** in the bottom pane
2. Click **Copy SQL connection string**
3. Extract the hostname (e.g., `abc123xyz.datawarehouse.fabric.microsoft.com`)

### 5. Credentials Configuration[​](#5-credentials-configuration "Direct link to 5. Credentials Configuration")

Both OneLake storage and Fabric publishing credentials go in `credentials.json`:

```
{
    "onelake_storage": {
        "ds_type": "onelake",
        "auth_mode": "service_principal",
        "info": {
            "directory": "onelake://your-workspace-name/your-lakehouse-name/",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    },
    "fabric_lakehouse": {
        "ds_type": "fabric",
        "auth_mode": "service_principal",
        "fabric_target": "lakehouse",
        "info": {
            "workspace_id": "your-workspace-id",
            "lakehouse_id": "your-lakehouse-id",
            "lakehouse_name": "your-lakehouse-name",
            "sql_endpoint": "your-sql-endpoint.datawarehouse.fabric.microsoft.com",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    }
}
```

**Configuration Fields:**

| Field                 | Description                     | Required |
| --------------------- | ------------------------------- | -------- |
| `ds_type`             | Must be `"fabric"`              | Yes      |
| `auth_mode`           | Must be `"service_principal"`   | Yes      |
| `fabric_target`       | Must be `"lakehouse"`           | Yes      |
| `workspace_id`        | Fabric workspace GUID           | Yes      |
| `lakehouse_id`        | Lakehouse GUID                  | Yes      |
| `lakehouse_name`      | Lakehouse display name          | Yes      |
| `sql_endpoint`        | SQL analytics endpoint hostname | Yes      |
| `azure_client_id`     | Application (client) ID         | Yes      |
| `azure_tenant_id`     | Directory (tenant) ID           | Yes      |
| `azure_client_secret` | Client secret value             | Yes      |

## Authentication Setup[​](#authentication-setup "Direct link to Authentication Setup")

Fabric authenticates via Service Principal (Azure AD).

### Service Principal Authentication[​](#service-principal-authentication "Direct link to Service Principal Authentication")

```
{
    "fabric_lakehouse": {
        "ds_type": "fabric",
        "auth_mode": "service_principal",
        "fabric_target": "lakehouse",
        "info": {
            "workspace_id": "your-workspace-id",
            "lakehouse_id": "your-lakehouse-id",
            "lakehouse_name": "your-lakehouse-name",
            "sql_endpoint": "your-sql-endpoint.datawarehouse.fabric.microsoft.com",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    }
}
```

### OneLake Directory Format[​](#onelake-directory-format "Direct link to OneLake Directory Format")

The OneLake directory must match your Fabric workspace and Lakehouse:

```
onelake://workspace-name/lakehouse-name/
```

Or using the Files path:

```
onelake://workspace-name/lakehouse-name.Lakehouse/Files/
```

## Table Types[​](#table-types "Direct link to Table Types")

LakeXpress supports two Fabric Lakehouse table modes.

### Internal Tables (Managed Delta Tables) - Recommended[​](#internal-tables-managed-delta-tables---recommended "Direct link to Internal Tables (Managed Delta Tables) - Recommended")

Delta tables in the Lakehouse Tables section. Data is converted to Delta Lake format with ACID transactions, time travel, and automatic SQL analytics endpoint availability.

* Full Delta Lake features (time travel, ACID, versioning)
* Optimized query performance
* Native Power BI integration
* Best for production workloads

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    ...

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

### External Tables (SQL Analytics Views)[​](#external-tables-sql-analytics-views "Direct link to External Tables (SQL Analytics Views)")

SQL views referencing Parquet files in the Files section. Data stays as Parquet; queries read files directly. Available via the SQL analytics endpoint only.

* No data conversion overhead
* Available immediately after export
* Suited for exploration and ad-hoc analysis
* Preserves original Parquet format

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method external \
    ...

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                | Default    |
| ---------------------------------- | ------------------------------------------ | ---------- |
| `--publish_target ID`              | Credential ID for Fabric target (required) | -          |
| `--publish_method METHOD`          | `internal` (Delta) or `external` (views)   | `internal` |
| `--publish_schema_pattern PATTERN` | Not applicable for Fabric (uses Lakehouse) | -          |
| `--publish_table_pattern PATTERN`  | Table naming pattern                       | `{table}`  |
| `--n_jobs N`                       | Parallel workers for table creation        | `1`        |

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                          | Example Output    |
| ------------- | ------------------------------------ | ----------------- |
| `{schema}`    | Source schema name                   | `tpch_1`          |
| `{table}`     | Source table name                    | `customer`        |
| `{database}`  | Source database name                 | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)              | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS) | `20251210_143022` |
| `{uuid}`      | UUID4 (consistent per run)           | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` parameter           | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### Prefixed Tables[​](#prefixed-tables "Direct link to Prefixed Tables")

```
./LakeXpress config create \
    --publish_table_pattern "src_{schema}_{table}" \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    ...

# Result: src_tpch_1_customer, src_tpch_1_orders
```

#### Schema-Based Naming[​](#schema-based-naming "Direct link to Schema-Based Naming")

```
./LakeXpress config create \
    --publish_table_pattern "{schema}_{table}" \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    ...

# Result: tpch_1_customer, tpch_1_orders
```

#### Date-Stamped Tables[​](#date-stamped-tables "Direct link to Date-Stamped Tables")

```
./LakeXpress config create \
    --publish_table_pattern "{table}_{date}" \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    ...

# Result: customer_20251210, orders_20251210
```

#### SQL Views with Prefix[​](#sql-views-with-prefix "Direct link to SQL Views with Prefix")

```
./LakeXpress config create \
    --publish_table_pattern "VW_{schema}_{table}" \
    --publish_target fabric_lakehouse \
    --publish_method external \
    ...

# Result: VW_tpch_1_customer, VW_tpch_1_orders
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: PostgreSQL to Fabric Delta Tables[​](#example-1-postgresql-to-fabric-delta-tables "Direct link to Example 1: PostgreSQL to Fabric Delta Tables")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

Tables (`customer`, `orders`, `lineitem`, etc.) appear in the Lakehouse Tables section, queryable via SQL analytics endpoint and Power BI.

### Example 2: SQL Server to Fabric with Custom Naming[​](#example-2-sql-server-to-fabric-with-custom-naming "Direct link to Example 2: SQL Server to Fabric with Custom Naming")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id mssql_prod \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --publish_table_pattern "dw_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

Creates tables `dw_DimCustomer`, `dw_DimProduct`, `dw_FactSales`, etc.

### Example 3: SQL Analytics Views (External)[​](#example-3-sql-analytics-views-external "Direct link to Example 3: SQL Analytics Views (External)")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method external \
    --publish_table_pattern "VW_{schema}_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

Parquet files land in the Files section; views `VW_tpch_1_customer`, `VW_tpch_1_orders`, etc. appear in the SQL analytics endpoint.

### Example 4: Parallel Table Creation[​](#example-4-parallel-table-creation "Direct link to Example 4: Parallel Table Creation")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 4 \
    --n_jobs 8 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

### Example 5: MySQL to Fabric[​](#example-5-mysql-to-fabric "Direct link to Example 5: MySQL to Fabric")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id mysql_prod \
    --source_db_name ecommerce \
    --source_schema_name ecommerce \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --publish_table_pattern "mysql_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id <SYNC_ID>
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Reference](/0.3/integrations/fabric-reference.md)Data type mappings, querying, troubleshooting, CLI arguments
* [![](/img/logos/onelake.png)Intermediate Storage](/0.3/configuration/storage/onelake.md)OneLake configuration
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# Microsoft Fabric Reference

Reference material for [Microsoft Fabric Publishing](/0.3/integrations/fabric-publishing.md) -- data type mappings, querying, troubleshooting, and CLI arguments.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types are mapped to Fabric-compatible Delta Lake types.

### PostgreSQL to Fabric[​](#postgresql-to-fabric "Direct link to PostgreSQL to Fabric")

| PostgreSQL Type      | Delta Lake Type |
| -------------------- | --------------- |
| `INTEGER`, `INT4`    | `INT`           |
| `BIGINT`, `INT8`     | `BIGINT`        |
| `SMALLINT`, `INT2`   | `SMALLINT`      |
| `NUMERIC(p,s)`       | `DECIMAL(p,s)`  |
| `REAL`, `FLOAT4`     | `FLOAT`         |
| `DOUBLE PRECISION`   | `DOUBLE`        |
| `VARCHAR(n)`, `TEXT` | `STRING`        |
| `DATE`               | `DATE`          |
| `TIMESTAMP`          | `TIMESTAMP`     |
| `TIMESTAMPTZ`        | `TIMESTAMP`     |
| `BOOLEAN`            | `BOOLEAN`       |
| `BYTEA`              | `BINARY`        |

### SQL Server to Fabric[​](#sql-server-to-fabric "Direct link to SQL Server to Fabric")

| SQL Server Type             | Delta Lake Type |
| --------------------------- | --------------- |
| `INT`                       | `INT`           |
| `BIGINT`                    | `BIGINT`        |
| `SMALLINT`                  | `SMALLINT`      |
| `TINYINT`                   | `TINYINT`       |
| `DECIMAL(p,s)`              | `DECIMAL(p,s)`  |
| `FLOAT`                     | `DOUBLE`        |
| `REAL`                      | `FLOAT`         |
| `VARCHAR(n)`, `NVARCHAR(n)` | `STRING`        |
| `DATE`                      | `DATE`          |
| `DATETIME`, `DATETIME2`     | `TIMESTAMP`     |
| `DATETIMEOFFSET`            | `TIMESTAMP`     |
| `BIT`                       | `BOOLEAN`       |
| `VARBINARY`                 | `BINARY`        |

## Querying Tables[​](#querying-tables "Direct link to Querying Tables")

### SQL Analytics Endpoint[​](#sql-analytics-endpoint "Direct link to SQL Analytics Endpoint")

```
-- Query a Delta table
SELECT * FROM your_lakehouse.dbo.customer LIMIT 10;

-- Aggregation
SELECT
    c_nationkey,
    COUNT(*) as customer_count,
    SUM(c_acctbal) as total_balance
FROM your_lakehouse.dbo.customer
GROUP BY c_nationkey
ORDER BY customer_count DESC;
```

### Spark Notebooks[​](#spark-notebooks "Direct link to Spark Notebooks")

```
# Read a Delta table
df = spark.read.table("customer")
df.show(10)

# Query with SQL
spark.sql("""
    SELECT c_nationkey, COUNT(*) as cnt
    FROM customer
    GROUP BY c_nationkey
""").show()

# Write results back
result_df.write.mode("overwrite").saveAsTable("customer_summary")
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

#### Authentication Errors[​](#authentication-errors "Direct link to Authentication Errors")

**Error:** `AADSTS7000215: Invalid client secret provided`

Regenerate your client secret in Azure AD and update `credentials.json`.

**Error:** `The user or service principal does not have access to the workspace`

1. Verify the Service Principal is added to the workspace
2. Ensure it has Member or Contributor role
3. Wait a few minutes for permissions to propagate

#### OneLake Connection Issues[​](#onelake-connection-issues "Direct link to OneLake Connection Issues")

**Error:** `Unable to connect to OneLake storage`

1. Verify the directory format: `onelake://workspace-name/lakehouse-name/`
2. Check that the Service Principal has Storage Blob Data Contributor role
3. Ensure workspace and lakehouse names are correct (case-sensitive)

#### Table Creation Failures[​](#table-creation-failures "Direct link to Table Creation Failures")

**Error:** `Failed to create table in Lakehouse`

1. Verify the SQL analytics endpoint is correct
2. Check that the Lakehouse is active (not paused)
3. Ensure sufficient capacity units are available
4. Confirm Parquet files were exported to OneLake

#### SQL Endpoint Connection Issues[​](#sql-endpoint-connection-issues "Direct link to SQL Endpoint Connection Issues")

**Error:** `Cannot connect to SQL analytics endpoint`

1. Verify the SQL endpoint hostname
2. Ensure the SQL analytics endpoint is enabled
3. Check firewall rules if connecting from on-premises

### Verifying Export Success[​](#verifying-export-success "Direct link to Verifying Export Success")

Check files in OneLake:

1. Open your Lakehouse in Fabric portal
2. Go to the **Files** section
3. Verify Parquet files exist in the expected folders

Check tables:

1. Open the **SQL analytics endpoint**
2. Expand **Tables** in the object explorer
3. Verify your tables appear

### Debug Mode[​](#debug-mode "Direct link to Debug Mode")

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID> \
    --log_lev DEBUG
```

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Fabric Publishing Arguments[​](#fabric-publishing-arguments "Direct link to Fabric Publishing Arguments")

| Option                            | Type    | Description                                         |
| --------------------------------- | ------- | --------------------------------------------------- |
| `--publish_target ID`             | String  | Credential ID for Fabric publishing (required)      |
| `--publish_method METHOD`         | Enum    | `internal` (Delta tables) or `external` (SQL views) |
| `--publish_table_pattern PATTERN` | String  | Table naming pattern (default: `{table}`)           |
| `--n_jobs N`                      | Integer | Parallel workers for table creation (default: 1)    |

### Full Example[​](#full-example "Direct link to Full Example")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_storage \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --publish_table_pattern "{schema}_{table}" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Publishing](/0.3/integrations/fabric-publishing.md)Setup and usage guide
* [![](/img/logos/onelake.png)Intermediate Storage](/0.3/configuration/storage/onelake.md)OneLake configuration
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# AWS Glue Publishing

LakeXpress creates **AWS Glue Data Catalog tables** from exported Parquet files, enabling queries via Athena, Redshift Spectrum, and EMR.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication Modes](#authentication-modes)
* [Configuration Options](#configuration-options)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. AWS Glue Permissions[​](#1-aws-glue-permissions "Direct link to 1. AWS Glue Permissions")

Required IAM permissions:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateDatabase",
                "glue:GetDatabase",
                "glue:DeleteDatabase",
                "glue:CreateTable",
                "glue:GetTable",
                "glue:DeleteTable",
                "glue:UpdateTable",
                "glue:GetTables",
                "glue:BatchCreatePartition",
                "glue:GetPartitions"
            ],
            "Resource": "*"
        }
    ]
}
```

### 2. S3 Storage Setup[​](#2-s3-storage-setup "Direct link to 2. S3 Storage Setup")

Both S3 and Glue credentials are required in `credentials.json`:

```
{
    "aws_s3_datalake": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://my-datalake-bucket/lakexpress/",
            "profile": "my-aws-profile"
        }
    },
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}
```

tip

Both `"aws_glue"` and `"glue"` are accepted as the `ds_type` value.

**Critical:** The S3 bucket must be accessible from the Glue Data Catalog in the specified region.

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

### Profile Authentication (Recommended)[​](#profile-authentication-recommended "Direct link to Profile Authentication (Recommended)")

Uses AWS CLI credentials from `~/.aws/credentials`:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}
```

* No secrets in config files
* Supports MFA and SSO profiles
* Easy credential rotation

### Keys Authentication[​](#keys-authentication "Direct link to Keys Authentication")

Uses explicit AWS access keys:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "keys",
        "info": {
            "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
            "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "region": "us-east-1"
        }
    }
}
```

Suited for CI/CD pipelines, containers, or cross-account access without AWS CLI.

### Role Authentication[​](#role-authentication "Direct link to Role Authentication")

Assumes an IAM role via STS:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "role",
        "info": {
            "role_arn": "arn:aws:iam::123456789012:role/GluePublishRole",
            "external_id": "optional-external-id",
            "region": "us-east-1"
        }
    }
}
```

Uses temporary credentials. Ideal for cross-account publishing and fine-grained access control.

## Understanding Glue Database Naming[​](#understanding-glue-database-naming "Direct link to Understanding Glue Database Naming")

No target database setting needed

Unlike traditional databases, the AWS Glue Data Catalog has no server or database instance to connect to. What Glue calls a "database" is simply a **namespace** — a container for organizing tables within the catalog. The catalog itself (scoped by `catalog_id`, which defaults to your AWS account) is the top-level container.

This is why there is no `--publish_database_name` option for Glue. Instead, `--publish_schema_pattern` directly controls the **Glue database name** that will be created. For example, `--publish_schema_pattern "datalake_{schema}"` creates a Glue database called `datalake_tpch_1`.

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                | Default    |
| ---------------------------------- | ------------------------------------------ | ---------- |
| `--publish_target ID`              | Credential ID for Glue target (required)   | -          |
| `--publish_schema_pattern PATTERN` | Glue database naming pattern               | `{schema}` |
| `--publish_table_pattern PATTERN`  | Table naming pattern                       | `{table}`  |
| `--glue_skip_existing`             | Skip existing tables instead of recreating | `false`    |
| `--n_jobs N`                       | Parallel workers for table creation        | `1`        |

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Database and table names support token-based patterns.

Token restrictions

The `{table}` token can only be used in `--publish_table_pattern`, not in `--publish_schema_pattern`. Since the schema pattern controls the Glue database name, using `{table}` there would attempt to create a separate database per table, which is not supported. LakeXpress validates this before starting the export and will report an error.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                          | Example Output    |
| ------------- | ------------------------------------ | ----------------- |
| `{schema}`    | Source schema name                   | `tpch_1`          |
| `{table}`     | Source table name                    | `customer`        |
| `{database}`  | Source database name                 | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)              | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS) | `20251210_143022` |
| `{uuid}`      | UUID4 (consistent per run)           | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` value               | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### Date-Partitioned Databases[​](#date-partitioned-databases "Direct link to Date-Partitioned Databases")

```
lakexpress \
  --publish_schema_pattern "lx_{schema}_{date}" \
  --publish_table_pattern "{table}" \
  --publish_target glue_catalog \
  ...

# Results:
# Database: lx_tpch_1_20251210
# Tables: customer, orders, lineitem
```

#### Prefixed Databases[​](#prefixed-databases "Direct link to Prefixed Databases")

```
lakexpress \
  --publish_schema_pattern "datalake_{schema}" \
  --publish_table_pattern "{table}" \
  --publish_target glue_catalog \
  ...

# Results:
# Database: datalake_tpch_1
# Tables: customer, orders, lineitem
```

#### Consolidated Multi-Schema[​](#consolidated-multi-schema "Direct link to Consolidated Multi-Schema")

```
lakexpress \
  --source_schema_name schema1,schema2 \
  --publish_schema_pattern "consolidated" \
  --publish_table_pattern "{schema}_{table}" \
  --publish_target glue_catalog \
  ...

# Results:
# Database: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: Basic Export to Glue[​](#example-1-basic-export-to-glue "Direct link to Example 1: Basic Export to Glue")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target glue_catalog
```

Creates database `public` with all tables from the `public` schema.

### Example 2: Custom Naming with Parallel Execution[​](#example-2-custom-naming-with-parallel-execution "Direct link to Example 2: Custom Naming with Parallel Execution")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target glue_catalog \
    --publish_schema_pattern "lx_{schema}" \
    --publish_table_pattern "{table}" \
    --n_jobs 4
```

Creates database `lx_tpch_1` with tables built in parallel.

### Example 3: Skip Existing Tables[​](#example-3-skip-existing-tables "Direct link to Example 3: Skip Existing Tables")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target glue_catalog \
    --glue_skip_existing
```

### Example 4: Date-Based Snapshots[​](#example-4-date-based-snapshots "Direct link to Example 4: Date-Based Snapshots")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name sales \
    --target_storage_id aws_s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target glue_catalog \
    --publish_schema_pattern "sales_{date}" \
    --sub_path "daily/$(date +%Y%m%d)"
```

Creates database `sales_20251210`, data stored at `s3://bucket/lakexpress/daily/20251210/`.

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/aws-glue.svg)AWS Glue Reference](/0.3/integrations/glue-reference.md)Data type mappings, CLI arguments, querying
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# AWS Glue Reference

Reference material for [AWS Glue Publishing](/0.3/integrations/glue-publishing.md) -- data type mappings, CLI arguments, and querying.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to Glue-compatible types.

### PostgreSQL to Glue[​](#postgresql-to-glue "Direct link to PostgreSQL to Glue")

| PostgreSQL Type      | Glue Type      |
| -------------------- | -------------- |
| `INTEGER`, `INT4`    | `int`          |
| `BIGINT`, `INT8`     | `bigint`       |
| `SMALLINT`, `INT2`   | `smallint`     |
| `NUMERIC(p,s)`       | `decimal(p,s)` |
| `REAL`, `FLOAT4`     | `float`        |
| `DOUBLE PRECISION`   | `double`       |
| `VARCHAR(n)`, `TEXT` | `string`       |
| `DATE`               | `date`         |
| `TIMESTAMP`          | `timestamp`    |
| `BOOLEAN`            | `boolean`      |
| `BYTEA`              | `binary`       |

### SQL Server to Glue[​](#sql-server-to-glue "Direct link to SQL Server to Glue")

| SQL Server Type             | Glue Type      |
| --------------------------- | -------------- |
| `INT`                       | `int`          |
| `BIGINT`                    | `bigint`       |
| `SMALLINT`                  | `smallint`     |
| `TINYINT`                   | `tinyint`      |
| `DECIMAL(p,s)`              | `decimal(p,s)` |
| `FLOAT`                     | `double`       |
| `REAL`                      | `float`        |
| `VARCHAR(n)`, `NVARCHAR(n)` | `string`       |
| `DATE`                      | `date`         |
| `DATETIME`, `DATETIME2`     | `timestamp`    |
| `BIT`                       | `boolean`      |
| `VARBINARY`                 | `binary`       |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Glue Publishing Arguments[​](#glue-publishing-arguments "Direct link to Glue Publishing Arguments")

| Option                             | Type    | Description                                       |
| ---------------------------------- | ------- | ------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for Glue publishing (required)      |
| `--publish_schema_pattern PATTERN` | String  | Database naming pattern (default: `{schema}`)     |
| `--publish_table_pattern PATTERN`  | String  | Table naming pattern (default: `{table}`)         |
| `--glue_skip_existing`             | Flag    | Skip existing tables instead of drop and recreate |
| `--n_jobs N`                       | Integer | Parallel workers for table creation (default: 1)  |

### Querying Glue Tables[​](#querying-glue-tables "Direct link to Querying Glue Tables")

**Amazon Athena:**

```
SELECT * FROM lx_tpch_1.customer LIMIT 10;
```

**AWS Redshift Spectrum:**

```
SELECT * FROM spectrum_schema.customer LIMIT 10;
```

**Amazon EMR (Spark):**

```
df = spark.table("lx_tpch_1.customer")
df.show(10)
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/aws-glue.svg)AWS Glue Publishing](/0.3/integrations/glue-publishing.md)Setup and usage guide
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# MotherDuck Publishing

LakeXpress creates MotherDuck tables from exported Parquet files in cloud storage (S3, GCS, Azure).

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication](#authentication)
* [Configuration Options](#configuration-options)
* [Table Types](#table-types)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. MotherDuck Account[​](#1-motherduck-account "Direct link to 1. MotherDuck Account")

Sign up at [motherduck.com](https://motherduck.com) and generate an access token:

1. Log in to MotherDuck
2. Go to Settings → Access Tokens
3. Create a new token
4. Copy the token for your credentials file

### 2. Cloud Storage Setup[​](#2-cloud-storage-setup "Direct link to 2. Cloud Storage Setup")

Configure credentials for cloud storage and MotherDuck:

```
{
    "s3_datalake": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://my-datalake-bucket/lakexpress/",
            "profile": "my-aws-profile"
        }
    },
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-access-token"
        }
    }
}
```

### 3. Storage Access from MotherDuck (Required for External Views)[​](#3-storage-access-from-motherduck-required-for-external-views "Direct link to 3. Storage Access from MotherDuck (Required for External Views)")

**Important:** When using external views (`--publish_method external`), MotherDuck needs credentials to read your Parquet files from cloud storage. Without this, you'll see errors like:

```
IO Error: No files found that match the pattern "s3://..."
HTTP Error: Permission error: Missing or invalid credentials
```

**Configure secrets in MotherDuck before querying external views:**

**For AWS S3 (most common):**

```
CREATE SECRET aws_s3_secret (
    TYPE S3,
    KEY_ID 'your_aws_access_key_id',
    SECRET 'your_aws_secret_access_key',
    REGION 'us-east-1'  -- your bucket's region
);
```

To find your AWS credentials:

```
aws configure get aws_access_key_id
aws configure get aws_secret_access_key
aws configure get region
```

**For GCS:**

```
CREATE SECRET gcs_secret (
    TYPE GCS,
    KEY_ID 'your_hmac_access_key',
    SECRET 'your_hmac_secret_key'
);
```

**For Azure:**

```
CREATE SECRET azure_secret (
    TYPE AZURE,
    ACCOUNT_NAME 'your_storage_account',
    ACCOUNT_KEY 'your_account_key'
);
```

**Verify secrets are configured:**

```
SELECT * FROM duckdb_secrets();
```

## Authentication[​](#authentication "Direct link to Authentication")

### Token Authentication (Recommended)[​](#token-authentication-recommended "Direct link to Token Authentication (Recommended)")

Store your token in the credentials file:

```
{
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-access-token"
        }
    }
}
```

| Field       | Required | Description                     |
| ----------- | -------- | ------------------------------- |
| `ds_type`   | Yes      | Must be `"motherduck"`          |
| `auth_mode` | Yes      | Set to `"token"`                |
| `database`  | Yes      | Target MotherDuck database name |
| `token`     | Yes      | MotherDuck access token         |

### Environment Variable Authentication[​](#environment-variable-authentication "Direct link to Environment Variable Authentication")

Read the token from an environment variable:

```
{
    "motherduck_env": {
        "ds_type": "motherduck",
        "auth_mode": "env",
        "info": {
            "database": "my_analytics"
        }
    }
}
```

```
export motherduck_token="your-motherduck-access-token"
lakexpress ...
```

| Field       | Required | Description                     |
| ----------- | -------- | ------------------------------- |
| `ds_type`   | Yes      | Must be `"motherduck"`          |
| `auth_mode` | Yes      | Set to `"env"`                  |
| `database`  | Yes      | Target MotherDuck database name |

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                    | Default    |
| ---------------------------------- | ---------------------------------------------- | ---------- |
| `--publish_target ID`              | Credential ID for MotherDuck target (required) | -          |
| `--publish_schema_pattern PATTERN` | Schema naming pattern                          | `{schema}` |
| `--publish_table_pattern PATTERN`  | Table naming pattern                           | `{table}`  |
| `--publish_method TYPE`            | Table type: `external` or `internal`           | `external` |
| `--n_jobs N`                       | Parallel workers for table creation            | `1`        |

## Table Types[​](#table-types "Direct link to Table Types")

### External Views (Default)[​](#external-views-default "Direct link to External Views (Default)")

Queries data directly from cloud storage via `read_parquet()`.

```
lakexpress ... --publish_target motherduck_prod --publish_method external
```

**Generated SQL:**

```
CREATE OR REPLACE VIEW "my_database"."tpch_1"."orders" AS
SELECT * FROM read_parquet('s3://bucket/exports/tpch_1/orders/*.parquet')
```

**Pros:**

* No data copying, instant publishing
* No MotherDuck storage costs
* Always reflects latest cloud storage data
* Best for large datasets, infrequent queries, exploration

**Trade-offs:**

* Query speed depends on cloud storage latency
* Requires cloud storage credentials in MotherDuck

### Internal Tables[​](#internal-tables "Direct link to Internal Tables")

Loads data into MotherDuck native columnar storage.

```
lakexpress ... --publish_target motherduck_prod --publish_method internal
```

**Generated SQL:**

```
CREATE OR REPLACE TABLE "my_database"."tpch_1"."orders" AS
SELECT * FROM read_parquet('s3://bucket/exports/tpch_1/orders/*.parquet')
```

**Pros:**

* Faster queries (optimized columnar format)
* No storage credentials needed after load
* Better caching for repeated queries
* Best for dashboards and production analytics

**Trade-offs:**

* Slower publishing (data must load)
* MotherDuck storage costs apply
* Data is a point-in-time snapshot

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                           | Example Output    |
| ------------- | ------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                    | `tpch_1`          |
| `{table}`     | Source table name                     | `customer`        |
| `{database}`  | Source database name                  | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)               | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  | `20251210_143022` |
| `{uuid}`      | UUID4 identifier (consistent per run) | `a1b2c3d4`        |
| `{subpath}`   | CLI `--sub_path` parameter            | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### Prefixed Schemas[​](#prefixed-schemas "Direct link to Prefixed Schemas")

```
lakexpress \
  --publish_schema_pattern "lx_{schema}" \
  --publish_table_pattern "{table}" \
  --publish_target motherduck_prod \
  ...

# Results:
# Schema: lx_tpch_1
# Tables: customer, orders, lineitem
```

#### Date-Partitioned Schemas[​](#date-partitioned-schemas "Direct link to Date-Partitioned Schemas")

```
lakexpress \
  --publish_schema_pattern "{schema}_{date}" \
  --publish_table_pattern "{table}" \
  --publish_target motherduck_prod \
  ...

# Results:
# Schema: tpch_1_20251210
# Tables: customer, orders, lineitem
```

#### Consolidated Multi-Schema[​](#consolidated-multi-schema "Direct link to Consolidated Multi-Schema")

```
lakexpress \
  --source_schema_name schema1,schema2 \
  --publish_schema_pattern "consolidated" \
  --publish_table_pattern "{schema}_{table}" \
  --publish_target motherduck_prod \
  ...

# Results:
# Schema: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: Basic Export with External Views[​](#example-1-basic-export-with-external-views "Direct link to Example 1: Basic Export with External Views")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target motherduck_prod
```

**Result:** Database `my_analytics`, schema `public`, external views over S3 Parquet files.

### Example 2: Internal Tables with Parallel Execution[​](#example-2-internal-tables-with-parallel-execution "Direct link to Example 2: Internal Tables with Parallel Execution")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name tpch_1 \
    --target_storage_id s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target motherduck_prod \
    --publish_method internal \
    --publish_schema_pattern "lx_{schema}" \
    --n_jobs 4
```

**Result:** Database `my_analytics`, schema `lx_tpch_1`, native tables loaded from S3.

### Example 3: Daily Snapshots[​](#example-3-daily-snapshots "Direct link to Example 3: Daily Snapshots")

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name sales \
    --target_storage_id s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target motherduck_prod \
    --publish_schema_pattern "sales_{date}" \
    --sub_path "daily/$(date +%Y%m%d)"
```

**Result:** Schema `sales_20251210`, data at `s3://bucket/lakexpress/daily/20251210/`.

### Example 4: Environment Variable Token[​](#example-4-environment-variable-token "Direct link to Example 4: Environment Variable Token")

```
export motherduck_token="your-token-here"

lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id s3_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target motherduck_env
```

### Example 5: GCS Storage Backend[​](#example-5-gcs-storage-backend "Direct link to Example 5: GCS Storage Backend")

```
{
    "gcs_datalake": {
        "ds_type": "gcs",
        "auth_mode": "profile",
        "info": {
            "directory": "gs://my-datalake-bucket/lakexpress/",
            "profile": "/path/to/service-account.json"
        }
    },
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-token"
        }
    }
}
```

```
lakexpress -a credentials.json \
    --lxdb_auth_id lxdb \
    --source_db_auth_id postgres_prod \
    --source_schema_name public \
    --target_storage_id gcs_datalake \
    --fastbcp_dir_path /path/to/FastBCP \
    --publish_target motherduck_prod
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/motherduck.svg)MotherDuck Reference](/0.3/integrations/motherduck-reference.md)Data type mappings, CLI arguments, querying, troubleshooting
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# MotherDuck Reference

Reference material for [MotherDuck Publishing](/0.3/integrations/motherduck-publishing.md) -- data type mappings, CLI arguments, querying, and troubleshooting.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map to DuckDB-compatible types automatically.

### PostgreSQL to DuckDB/MotherDuck[​](#postgresql-to-duckdbmotherduck "Direct link to PostgreSQL to DuckDB/MotherDuck")

| PostgreSQL Type      | DuckDB Type    |
| -------------------- | -------------- |
| `INTEGER`, `INT4`    | `INTEGER`      |
| `BIGINT`, `INT8`     | `BIGINT`       |
| `SMALLINT`, `INT2`   | `SMALLINT`     |
| `NUMERIC(p,s)`       | `DECIMAL(p,s)` |
| `REAL`, `FLOAT4`     | `REAL`         |
| `DOUBLE PRECISION`   | `DOUBLE`       |
| `VARCHAR(n)`, `TEXT` | `VARCHAR`      |
| `DATE`               | `DATE`         |
| `TIMESTAMP`          | `TIMESTAMP`    |
| `TIMESTAMPTZ`        | `TIMESTAMPTZ`  |
| `TIME`               | `TIME`         |
| `BOOLEAN`            | `BOOLEAN`      |
| `BYTEA`              | `BLOB`         |
| `JSON`, `JSONB`      | `JSON`         |
| `UUID`               | `UUID`         |

### SQL Server to DuckDB/MotherDuck[​](#sql-server-to-duckdbmotherduck "Direct link to SQL Server to DuckDB/MotherDuck")

| SQL Server Type             | DuckDB Type     |
| --------------------------- | --------------- |
| `INT`                       | `INTEGER`       |
| `BIGINT`                    | `BIGINT`        |
| `SMALLINT`                  | `SMALLINT`      |
| `TINYINT`                   | `TINYINT`       |
| `DECIMAL(p,s)`              | `DECIMAL(p,s)`  |
| `MONEY`                     | `DECIMAL(19,4)` |
| `FLOAT`                     | `DOUBLE`        |
| `REAL`                      | `REAL`          |
| `VARCHAR(n)`, `NVARCHAR(n)` | `VARCHAR`       |
| `DATE`                      | `DATE`          |
| `DATETIME`, `DATETIME2`     | `TIMESTAMP`     |
| `DATETIMEOFFSET`            | `TIMESTAMPTZ`   |
| `TIME`                      | `TIME`          |
| `BIT`                       | `BOOLEAN`       |
| `VARBINARY`                 | `BLOB`          |
| `UNIQUEIDENTIFIER`          | `UUID`          |

### Oracle to DuckDB/MotherDuck[​](#oracle-to-duckdbmotherduck "Direct link to Oracle to DuckDB/MotherDuck")

| Oracle Type             | DuckDB Type     |
| ----------------------- | --------------- |
| `NUMBER`                | `DECIMAL(38,9)` |
| `NUMBER(p,s)`           | `DECIMAL(p,s)`  |
| `FLOAT`                 | `DOUBLE`        |
| `VARCHAR2`, `NVARCHAR2` | `VARCHAR`       |
| `CLOB`, `NCLOB`         | `VARCHAR`       |
| `DATE`                  | `TIMESTAMP`     |
| `TIMESTAMP`             | `TIMESTAMP`     |
| `RAW`, `BLOB`           | `BLOB`          |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### MotherDuck Publishing Arguments[​](#motherduck-publishing-arguments "Direct link to MotherDuck Publishing Arguments")

| Option                             | Type    | Description                                                |
| ---------------------------------- | ------- | ---------------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for MotherDuck publishing (required)         |
| `--publish_schema_pattern PATTERN` | String  | Dynamic schema naming pattern (default: `{schema}`)        |
| `--publish_table_pattern PATTERN`  | String  | Dynamic table naming pattern (default: `{table}`)          |
| `--publish_method TYPE`            | String  | Table type: `external` (default) or `internal`             |
| `--n_jobs N`                       | Integer | Number of parallel workers for table creation (default: 1) |

### Querying MotherDuck Tables[​](#querying-motherduck-tables "Direct link to Querying MotherDuck Tables")

Once published, query tables via any of these methods.

**MotherDuck Web UI:**

Log in to [app.motherduck.com](https://app.motherduck.com) and run:

```
SELECT * FROM my_analytics.lx_tpch_1.customer LIMIT 10;
```

**DuckDB CLI:**

```
duckdb "md:my_analytics?motherduck_token=your_token"
```

```
SELECT * FROM lx_tpch_1.customer LIMIT 10;
```

**Python (duckdb):**

```
import duckdb

conn = duckdb.connect("md:my_analytics?motherduck_token=your_token")
df = conn.execute("SELECT * FROM lx_tpch_1.customer LIMIT 10").df()
print(df)
```

**Python with environment variable:**

```
import os
import duckdb

os.environ["motherduck_token"] = "your_token"
conn = duckdb.connect("md:my_analytics")
df = conn.execute("SELECT * FROM lx_tpch_1.customer LIMIT 10").df()
print(df)
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

**"Authentication failed":**

* Verify your token is valid and not expired
* Check token permissions for the target database
* For env mode, ensure `motherduck_token` is set

**"Database not found":**

* Verify the database name matches an existing MotherDuck database
* Create the database in MotherDuck UI before publishing

**"IO Error: No files found that match the pattern" or "Permission error: Missing or invalid credentials":**

* This means MotherDuck cannot access your private S3/GCS/Azure bucket
* Create a secret in MotherDuck with your cloud storage credentials (see [Storage Access](/0.3/integrations/motherduck-publishing.md#3-storage-access-from-motherduck-required-for-external-views))
* Verify secrets are configured: `SELECT * FROM duckdb_secrets();`
* Test access: `SELECT count(*) FROM read_parquet('s3://your-bucket/path/*.parquet');`

**"Cannot read file" (external views):**

* Verify cloud storage credentials in MotherDuck
* Check that the S3/GCS/Azure path contains valid Parquet files
* Ensure the bucket allows access from MotherDuck
* Verify the files exist: `aws s3 ls s3://your-bucket/path/ --recursive | head`

**"Connection timeout":**

* Check network connectivity
* MotherDuck requires outbound HTTPS
* Verify no firewall is blocking the connection

### Verifying Setup[​](#verifying-setup "Direct link to Verifying Setup")

Test MotherDuck connectivity:

```
duckdb "md:my_analytics?motherduck_token=your_token" -c "SELECT 1"
```

Test cloud storage access from MotherDuck:

```
SELECT * FROM read_parquet('s3://your-bucket/path/to/file.parquet') LIMIT 1;
```

### Configuring Storage Secrets[​](#configuring-storage-secrets "Direct link to Configuring Storage Secrets")

See [Storage Access from MotherDuck](/0.3/integrations/motherduck-publishing.md#3-storage-access-from-motherduck-required-for-external-views) in Prerequisites for detailed instructions on configuring S3, GCS, or Azure secrets.

## Comparison with Other Targets[​](#comparison-with-other-targets "Direct link to Comparison with Other Targets")

| Feature         | MotherDuck                  | BigQuery              | Snowflake                 | DuckLake          |
| --------------- | --------------------------- | --------------------- | ------------------------- | ----------------- |
| Query Engine    | DuckDB                      | BigQuery              | Snowflake                 | DuckDB            |
| External Tables | Views with `read_parquet()` | External tables       | External tables           | DuckLake catalog  |
| Native Tables   | CREATE TABLE AS SELECT      | Load jobs             | COPY INTO                 | Not supported     |
| Serverless      | Yes                         | Yes                   | No (warehouse required)   | N/A (local)       |
| SQL Dialect     | DuckDB/PostgreSQL           | BigQuery SQL          | Snowflake SQL             | DuckDB/PostgreSQL |
| Best For        | Analytics, exploration      | Large-scale analytics | Enterprise data warehouse | Local development |

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/motherduck.svg)MotherDuck Publishing](/0.3/integrations/motherduck-publishing.md)Setup and usage guide
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# Amazon Redshift Publishing

LakeXpress **creates Amazon Redshift tables** from exported Parquet files stored in S3, supporting both native internal tables and Spectrum external tables.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication Modes](#authentication-modes)
* [Table Types](#table-types)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)
* [Primary Key Constraints](#primary-key-constraints)
* [Notes and Limitations](#notes-and-limitations)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Redshift Cluster[​](#1-redshift-cluster "Direct link to 1. Redshift Cluster")

A Redshift Provisioned cluster or Redshift Serverless workgroup with:

* A database (default: `dev`)
* A user with `CREATE SCHEMA` and `CREATE TABLE` privileges
* Security group allowing inbound TCP on port 5439

### 2. IAM Role[​](#2-iam-role "Direct link to 2. IAM Role")

An IAM role that Redshift can assume, with:

* **S3 read access** on the bucket containing exported Parquet files (required for both internal and external tables)
* **AWS Glue catalog access** (required only for external/Spectrum tables)

Example trust policy:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "redshift.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}
```

### 3. S3 Storage[​](#3-s3-storage "Direct link to 3. S3 Storage")

Both the Redshift cluster and the S3 bucket must be in the **same AWS region** to avoid cross-region transfer costs.

The S3 credential and the Redshift credential must both be in your `credentials.json`:

```
{
  "aws_s3": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-bucket/lakexpress/",
      "profile_name": "lakexpress"
    }
  },
  "redshift_classic": {
    "ds_type": "redshift",
    "auth_mode": "classic",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "username": "$env{LX_REDSHIFT_USER}",
      "password": "$env{LX_REDSHIFT_PASSWORD}",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

### Classic (Username/Password)[​](#classic-usernamepassword "Direct link to Classic (Username/Password)")

```
{
  "redshift_classic": {
    "ds_type": "redshift",
    "auth_mode": "classic",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "username": "$env{LX_REDSHIFT_USER}",
      "password": "$env{LX_REDSHIFT_PASSWORD}",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

| Field      | Required | Description                       |
| ---------- | -------- | --------------------------------- |
| `host`     | Yes      | Cluster endpoint                  |
| `port`     | No       | Default: 5439                     |
| `database` | No       | Default: `dev`                    |
| `username` | Yes      | Database user                     |
| `password` | Yes      | Database password                 |
| `iam_role` | Yes      | IAM role ARN for S3/Glue access   |
| `region`   | No       | AWS region (default: `us-east-1`) |

### IAM Authentication[​](#iam-authentication "Direct link to IAM Authentication")

```
{
  "redshift_iam": {
    "ds_type": "redshift",
    "auth_mode": "iam",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "cluster_identifier": "my-cluster",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

| Field                | Required | Description                       |
| -------------------- | -------- | --------------------------------- |
| `cluster_identifier` | Yes      | Redshift cluster identifier       |
| `iam_role`           | Yes      | IAM role ARN                      |
| `region`             | No       | AWS region (default: `us-east-1`) |

## Table Types[​](#table-types "Direct link to Table Types")

### Internal Tables (`--publish_method internal`)[​](#internal-tables---publish_method-internal "Direct link to internal-tables---publish_method-internal")

Creates native Redshift tables and loads data using `COPY FROM S3 FORMAT AS PARQUET`.

* Data is **copied into Redshift storage** -- queries are fast but storage costs apply
* The COPY command reads all Parquet files in the S3 directory (including partitioned exports)
* Decimal columns are mapped to `DECIMAL(38,18)` to match the Parquet physical layout
* Supports `--pk_constraints` for informational primary key constraints

### External Tables (`--publish_method external`)[​](#external-tables---publish_method-external "Direct link to external-tables---publish_method-external")

Creates Redshift Spectrum tables backed by the AWS Glue Data Catalog.

* Data **stays in S3** -- no data movement, no Redshift storage costs
* Requires an IAM role with Glue catalog permissions
* The Glue database is created automatically (`CREATE EXTERNAL DATABASE IF NOT EXISTS`)
* Decimal columns preserve source precision (e.g., `DECIMAL(12,2)`)
* PK constraints are not supported for external tables

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

| Token       | Description             |
| ----------- | ----------------------- |
| `{schema}`  | Source schema name      |
| `{table}`   | Source table name       |
| `{subpath}` | Value of `--sub_path`   |
| `{date}`    | Current date (YYYYMMDD) |

Set via `--publish_schema_pattern` and `--publish_table_pattern`.

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Internal Tables[​](#internal-tables "Direct link to Internal Tables")

```
LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id log_db \
  --source_db_auth_id source_db \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id aws_s3 \
  --sub_path mydata \
  --publish_target redshift_classic \
  --publish_method internal \
  --publish_schema_pattern "INT_{subpath}_{date}" \
  --publish_table_pattern "{schema}_{table}"
```

### External Tables (Spectrum)[​](#external-tables-spectrum "Direct link to External Tables (Spectrum)")

```
LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id log_db \
  --source_db_auth_id source_db \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id aws_s3 \
  --sub_path mydata \
  --publish_target redshift_classic \
  --publish_method external \
  --publish_schema_pattern "EXT_{subpath}_{date}" \
  --publish_table_pattern "{schema}_{table}"
```

### With Primary Key Constraints[​](#with-primary-key-constraints "Direct link to With Primary Key Constraints")

Add `--pk_constraints` to include primary key definitions in internal table DDL:

```
LakeXpress config create \
  ... \
  --publish_method internal \
  --publish_target redshift_classic \
  --pk_constraints
```

## Primary Key Constraints[​](#primary-key-constraints "Direct link to Primary Key Constraints")

When `--pk_constraints` is set, LakeXpress adds `PRIMARY KEY` constraints to internal table DDL based on the source table's primary key columns.

These constraints are **informational only** in Redshift -- they are not enforced but are used by the query optimizer for better execution plans.

PK constraints are not supported for external (Spectrum) tables.

## Notes and Limitations[​](#notes-and-limitations "Direct link to Notes and Limitations")

* **COPY parallelism**: The COPY command automatically distributes file reads across Redshift compute slices. Multi-file exports (partitioned tables) benefit from multi-node clusters.
* **Decimal mapping**: Internal tables use `DECIMAL(38,18)` because Redshift COPY requires the column byte size to exactly match the Parquet physical type. External tables preserve source precision.
* **Region**: The Redshift cluster and S3 bucket should be in the same AWS region to avoid cross-region data transfer costs.
* **Glue database**: For external tables, the `CREATE EXTERNAL SCHEMA` DDL automatically creates the backing Glue database if it does not exist.
* **Redshift Serverless**: Both Provisioned and Serverless clusters are supported. The connection uses the same host/port/user/password parameters.
* **Driver**: Requires `redshift_connector`. Install with `pip install redshift_connector`.

## See Also[​](#see-also "Direct link to See Also")

* [Compatibility](/0.3/compatibility.md)Full list of supported platforms
* [![](/img/logos/aws-s3.png)Storage Configuration: S3](/0.3/configuration/storage/s3.md)S3 credential setup


---

# Snowflake Publishing

LakeXpress **creates Snowflake tables** from exported Parquet files, enabling queries on data stored in S3, Azure, or GCS.

## Table of Contents[​](#table-of-contents "Direct link to Table of Contents")

* [Prerequisites](#prerequisites)
* [Authentication Modes](#authentication-modes)
* [Table Types](#table-types)
* [Dynamic Naming Patterns](#dynamic-naming-patterns)
* [Usage Examples](#usage-examples)
* [Views and Customization](#views-and-customization)
* [Primary Key Constraints](#primary-key-constraints)

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Snowflake Setup[​](#1-snowflake-setup "Direct link to 1. Snowflake Setup")

**Required objects:**

* Database (e.g., `LAKEXPRESS_DB`)
* Warehouse (e.g., `LAKEXPRESS_WH`)
* Stage pointing to S3 bucket (e.g., `LAKEXPRESS_STAGE`)

**Required privileges:**

```
USE ROLE ACCOUNTADMIN;
GRANT CREATE SCHEMA ON DATABASE LAKEXPRESS_DB TO ROLE SYSADMIN;
GRANT USAGE ON WAREHOUSE LAKEXPRESS_WH TO ROLE SYSADMIN;
GRANT USAGE ON STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE TO ROLE SYSADMIN;
```

### 2. Cloud Storage Setup[​](#2-cloud-storage-setup "Direct link to 2. Cloud Storage Setup")

Both S3 and Snowflake credentials must be in your `credentials.json`:

```
{
  "aws_s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-datalake-bucket/lakexpress/",
      "profile": "lakexpress"
    }
  },
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "password",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "password": "$env{LX_SNOWFLAKE_PASSWORD}",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE"
    }
  }
}
```

**Critical:** The S3 `directory` path **must exactly match** the Snowflake stage location.

**Verify the match:**

```
DESC STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE;
-- URL column shows: s3://my-datalake-bucket/lakexpress/

-- This MUST match your S3 credentials directory
-- "directory": "s3://my-datalake-bucket/lakexpress/"  ✓ Correct
-- "directory": "s3://my-datalake-bucket/"              ✗ Wrong - files won't be found
```

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

### Password Authentication[​](#password-authentication "Direct link to Password Authentication")

```
{
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "password",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "password": "$env{LX_SNOWFLAKE_PASSWORD}",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE"
    }
  }
}
```

### Programmatic Access Token (PAT) Authentication[​](#programmatic-access-token-pat-authentication "Direct link to Programmatic Access Token (PAT) Authentication")

Recommended for automation:

```
{
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "pat",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "token": "your-personal-access-token",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE"
    }
  }
}
```

PATs require a network policy configured for the user.

**PAT benefits:**

* Easily rotated, configurable lifetime
* No MFA prompts -- suited for CI/CD pipelines

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables[​](#external-tables "Direct link to External Tables")

Data stays in cloud storage; Snowflake stores only metadata. Views are auto-created for cleaner querying.

* No data loading time or Snowflake storage costs
* Immediate availability after export
* Suited for: data lake integration, infrequent access, exploration, ETL staging

```
./LakeXpress \
  -a data/ds_credentials.json \
  --source_db_auth_id ds_04_pg \
  --target_storage_id aws_s3_01 \
  --publish_target snowflake_prod \
  --snowflake_table_type external  # Default
```

### Internal Tables[​](#internal-tables "Direct link to Internal Tables")

Data is loaded into Snowflake storage with full feature support (clustering, time travel, etc.). Can include PRIMARY KEY constraints from source.

* Faster queries with full Snowflake features
* Maintains PRIMARY KEY constraints
* Suited for: frequently queried data, production warehouses, complex joins

```
./LakeXpress \
  -a data/ds_credentials.json \
  --source_db_auth_id ds_04_pg \
  --target_storage_id aws_s3_01 \
  --publish_target snowflake_prod \
  --snowflake_table_type internal \
  --snowflake_pk_constraints  # Optional: include PK constraints
```

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns for time-partitioned exports, schema consolidation, or unique identifiers.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                           | Example Output    |
| ------------- | ------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                    | `tpch_1`          |
| `{table}`     | Source table name                     | `customer`        |
| `{database}`  | Source database name                  | `tpch`            |
| `{date}`      | Current date (YYYYMMDD)               | `20251120`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  | `20251120_112511` |
| `{uuid}`      | UUID4 identifier (consistent per run) | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` parameter            | `toto`            |

**Note:** All timestamps are in local time. The `{table}` token is **mandatory** for `--publish_table_pattern`.

### CLI Arguments[​](#cli-arguments "Direct link to CLI Arguments")

```
--publish_schema_pattern PATTERN    # Schema name pattern (default varies by table type)
--publish_table_pattern PATTERN      # Table name pattern (must include {table} token)
```

### Default Patterns[​](#default-patterns "Direct link to Default Patterns")

| Table Type | Default Schema Pattern | Default Table Pattern |
| ---------- | ---------------------- | --------------------- |
| External   | `EXT_{schema}`         | `{table}`             |
| Internal   | `{schema}`             | `{table}`             |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

#### 1. Time-Based Partitioning[​](#1-time-based-partitioning "Direct link to 1. Time-Based Partitioning")

```
./LakeXpress \
  --publish_schema_pattern "EXT_{schema}_{date}" \
  --publish_table_pattern "{table}" \
  --publish_target snowflake_prod \
  ...

# Results:
# Schema: EXT_tpch_1_20251120
# Tables: customer, orders, lineitem
```

#### 2. Multi-Schema Consolidation[​](#2-multi-schema-consolidation "Direct link to 2. Multi-Schema Consolidation")

```
./LakeXpress \
  --source_schema_name schema1,schema2,schema3 \
  --publish_schema_pattern "{subpath}" \
  --publish_table_pattern "{schema}_{table}" \
  --sub_path consolidated \
  --publish_target snowflake_prod \
  ...

# Results:
# Schema: consolidated
# Tables: schema1_customer, schema2_customer, schema3_customer
```

#### 3. Unique Run Identifiers[​](#3-unique-run-identifiers "Direct link to 3. Unique Run Identifiers")

```
./LakeXpress \
  --publish_schema_pattern "EXT_{schema}_{uuid}" \
  --publish_table_pattern "{table}" \
  --publish_target snowflake_prod \
  ...

# Results:
# Schema: EXT_tpch_1_a1b2c3d4-e5f6-7890-abcd-ef1234567890
# Tables: customer, orders, lineitem
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

### Example 1: External Tables with Custom Patterns[​](#example-1-external-tables-with-custom-patterns "Direct link to Example 1: External Tables with Custom Patterns")

```
./LakeXpress -a data/ds_credentials.json \
       --lxdb_auth_id lxdb_ms \
       --source_db_auth_id ds_04_pg \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_dir_path /path/to/FastBCP/ \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id aws_s3_01 \
       --generate_metadata \
       --sub_path toto \
       --publish_target snowflake_pass \
       --publish_schema_pattern "EXT_{subpath}_{date}" \
       --publish_table_pattern "{schema}_{table}" \
       --snowflake_table_type external
```

**Result:**

* Schema: `EXT_toto_20251124`
* Tables: `tpch_1_customer`, `tpch_1_orders`, etc.
* Views: `V_tpch_1_customer`, `V_tpch_1_orders`, etc.

### Example 2: Internal Tables with Primary Keys[​](#example-2-internal-tables-with-primary-keys "Direct link to Example 2: Internal Tables with Primary Keys")

```
./LakeXpress -a data/ds_credentials.json \
       --lxdb_auth_id lxdb_ms \
       --source_db_auth_id ds_04_pg \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_dir_path /path/to/FastBCP/ \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id aws_s3_01 \
       --generate_metadata \
       --sub_path toto \
       --snowflake_table_type internal \
       --publish_target snowflake_pat \
       --snowflake_pk_constraints
```

**Result:**

* Schema: `tpch_1`
* Tables: `customer`, `orders`, etc. (with data loaded)
* PRIMARY KEY constraints applied where available

### Example 3: Single Table[​](#example-3-single-table "Direct link to Example 3: Single Table")

```
./LakeXpress \
  -a data/ds_credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id ds_04_pg \
  --target_storage_id aws_s3_01 \
  --source_schema_name tpch_1 \
  -i nation \
  --fastbcp_dir_path /path/to/FastBCP \
  --publish_target snowflake_prod
```

### Example 4: Full Schema[​](#example-4-full-schema "Direct link to Example 4: Full Schema")

```
./LakeXpress \
  -a data/ds_credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id ds_04_pg \
  --target_storage_id aws_s3_01 \
  --source_schema_name tpch_1 \
  --fastbcp_dir_path /path/to/FastBCP \
  --publish_target snowflake_prod
```

## Views and Customization[​](#views-and-customization "Direct link to Views and Customization")

### Automatic Views (External Tables Only)[​](#automatic-views-external-tables-only "Direct link to Automatic Views (External Tables Only)")

For external tables, LakeXpress auto-creates views that expose typed columns instead of the raw Parquet `VALUE` column.

```
-- External table (raw)
SELECT * FROM EXT_TPCH_1.CUSTOMER;
-- Shows: VALUE column (complex structure)

-- View (typed columns)
SELECT * FROM EXT_TPCH_1.V_CUSTOMER;
-- Shows: C_CUSTKEY, C_NAME, C_ADDRESS, etc.
```

### View Options[​](#view-options "Direct link to View Options")

| Option                           | Description             | Default       |
| -------------------------------- | ----------------------- | ------------- |
| `--snowflake_no_views`           | Skip view creation      | Views created |
| `--snowflake_view_prefix PREFIX` | Custom view name prefix | `V_`          |

**Custom prefix:**

```
./LakeXpress \
  --publish_target snowflake_prod \
  --snowflake_table_type external \
  --snowflake_view_prefix "VIEW_" \
  ...

# Tables: customer, orders
# Views: VIEW_customer, VIEW_orders
```

**No views:**

```
./LakeXpress \
  --publish_target snowflake_prod \
  --snowflake_table_type external \
  --snowflake_no_views \
  ...

# Tables: customer, orders
# Views: None
```

### Refreshing External Tables[​](#refreshing-external-tables "Direct link to Refreshing External Tables")

External tables must be refreshed after data upload:

```
ALTER EXTERNAL TABLE EXT_TPCH_1.CUSTOMER REFRESH;
ALTER EXTERNAL TABLE EXT_TPCH_1.LINEITEM REFRESH;
```

Or via Snowflake CLI:

```
snow sql -q "ALTER EXTERNAL TABLE EXT_TPCH_1.LINEITEM REFRESH;" -c lakexpress
```

## Primary Key Constraints[​](#primary-key-constraints "Direct link to Primary Key Constraints")

Applies to **internal tables** only. LakeXpress propagates PRIMARY KEY constraints from the source database to Snowflake.

### Usage[​](#usage "Direct link to Usage")

```
./LakeXpress \
  --publish_target snowflake_prod \
  --snowflake_table_type internal \
  --snowflake_pk_constraints \
  ...
```

### How It Works[​](#how-it-works "Direct link to How It Works")

1. Reads PRIMARY KEY metadata from the source database
2. Stores PK information in the LakeXpress DB
3. Adds PRIMARY KEY constraints to the Snowflake DDL
4. Snowflake enforces constraints (query planner benefits)

### Requirements[​](#requirements "Direct link to Requirements")

* `--snowflake_table_type internal` only
* Source database must have PRIMARY KEY constraints defined
* LakeXpress DB must have `is_primary_key` column (auto-created in v0.1.16+)

### Benefits[​](#benefits "Direct link to Benefits")

* Query planner optimizations
* Better execution plans
* Documents data integrity

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/snowflake.svg)Snowflake Reference](/0.3/integrations/snowflake-reference.md)Data type mappings, CLI arguments, deprecated options
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options
* [Examples & Recipes](/0.3/guides/command-cookbook.md)Working command examples


---

# Snowflake Reference

Reference material for [Snowflake Publishing](/0.3/integrations/snowflake-publishing.md) -- data type mappings and CLI arguments.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

LakeXpress maps source types to Snowflake types automatically.

### PostgreSQL to Snowflake[​](#postgresql-to-snowflake "Direct link to PostgreSQL to Snowflake")

| PostgreSQL Type    | Snowflake Type      |
| ------------------ | ------------------- |
| `INTEGER`, `INT4`  | `NUMBER(38,0)`      |
| `BIGINT`, `INT8`   | `NUMBER(38,0)`      |
| `SMALLINT`, `INT2` | `NUMBER(38,0)`      |
| `NUMERIC(p,s)`     | `NUMBER(p,s)`       |
| `REAL`, `FLOAT4`   | `FLOAT`             |
| `DOUBLE PRECISION` | `DOUBLE`            |
| `VARCHAR(n)`       | `VARCHAR(n)`        |
| `TEXT`             | `VARCHAR(16777216)` |
| `DATE`             | `DATE`              |
| `TIMESTAMP`        | `TIMESTAMP_NTZ`     |
| `BOOLEAN`          | `BOOLEAN`           |
| `JSON`, `JSONB`    | `VARIANT`           |

### SQL Server to Snowflake[​](#sql-server-to-snowflake "Direct link to SQL Server to Snowflake")

| SQL Server Type         | Snowflake Type      |
| ----------------------- | ------------------- |
| `INT`                   | `NUMBER(38,0)`      |
| `BIGINT`                | `NUMBER(38,0)`      |
| `DECIMAL(p,s)`          | `NUMBER(p,s)`       |
| `FLOAT`                 | `DOUBLE`            |
| `NVARCHAR(n)`           | `VARCHAR(n)`        |
| `VARCHAR(MAX)`          | `VARCHAR(16777216)` |
| `DATE`                  | `DATE`              |
| `DATETIME`, `DATETIME2` | `TIMESTAMP_NTZ`     |
| `BIT`                   | `BOOLEAN`           |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Snowflake Publishing Arguments[​](#snowflake-publishing-arguments "Direct link to Snowflake Publishing Arguments")

| Option                             | Type   | Description                                                                           |
| ---------------------------------- | ------ | ------------------------------------------------------------------------------------- |
| `--publish_target ID`              | String | Credential ID for Snowflake publishing                                                |
| `--publish_schema_pattern PATTERN` | String | Schema naming pattern (default: `EXT_{schema}` for external, `{schema}` for internal) |
| `--publish_table_pattern PATTERN`  | String | Table naming pattern (default: `{table}`)                                             |
| `--snowflake_table_type TYPE`      | Enum   | `external` or `internal` (default: `external`)                                        |
| `--snowflake_no_views`             | Flag   | Skip view creation (external tables only)                                             |
| `--snowflake_view_prefix PREFIX`   | String | View name prefix (default: `V_`)                                                      |
| `--snowflake_pk_constraints`       | Flag   | Add PRIMARY KEY constraints (internal tables only)                                    |

### Deprecated Arguments[​](#deprecated-arguments "Direct link to Deprecated Arguments")

| Old Argument | New Argument       | Notes               |
| ------------ | ------------------ | ------------------- |
| `--publish`  | `--publish_target` | Removed in v0.1.12+ |

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/snowflake.svg)Snowflake Publishing](/0.3/integrations/snowflake-publishing.md)Setup and usage guide
* [CLI Reference](/0.3/reference/cli-reference.md)All command-line options


---

# CLI Reference

All LakeXpress commands and options.

## Overview[​](#overview "Direct link to Overview")

LakeXpress uses a two-step workflow:

1. **Create a sync configuration** with `LakeXpress config create` -- stores settings in database
2. **Execute the sync** with `LakeXpress sync` -- runs the export

Additional commands: `lxdb` for database management, `status` for monitoring.

## Basic Usage[​](#basic-usage "Direct link to Basic Usage")

```
LakeXpress [COMMAND] [OPTIONS]
```

## Global Options[​](#global-options "Direct link to Global Options")

### Help and Version[​](#help-and-version "Direct link to Help and Version")

| Option          | Description                                        |
| --------------- | -------------------------------------------------- |
| `-h, --help`    | Show help message and exit                         |
| `--version, -v` | Show version                                       |
| `--no_banner`   | Suppress startup banner                            |
| `--no_progress` | Disable progress bar                               |
| `--quiet_fbcp`  | Suppress FastBCP log output (stats still recorded) |

### License Options[​](#license-options "Direct link to License Options")

| Option                | Type   | Description                                                |
| --------------------- | ------ | ---------------------------------------------------------- |
| `--license TEXT`      | String | License text (alternative to `LAKEXPRESS_LICENSE` env var) |
| `--license_file PATH` | String | Path to license file                                       |

License Resolution Order

LakeXpress checks for a license in the following order:

1. `LAKEXPRESS_LICENSE` environment variable

2. `--license` CLI argument

3. `--license_file` CLI argument

4. Default file locations:

   <!-- -->

   * **Linux/Mac:** `~/.lakexpress/license.lic`
   * **Windows:** `%APPDATA%/LakeXpress/license.lic`
   * **Current directory:** `./license.lic`

5. Trial mode (automatic 30-day trial)

## Database Lifecycle Management[​](#database-lifecycle-management "Direct link to Database Lifecycle Management")

### lxdb init[​](#lxdb-init "Direct link to lxdb init")

Initialize the LakeXpress DB schema.

```
./LakeXpress lxdb init \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |

**When to use:** Optional -- the schema auto-creates on first sync. Useful to pre-verify connectivity.

### lxdb drop[​](#lxdb-drop "Direct link to lxdb drop")

Drop the LakeXpress DB schema.

```
./LakeXpress lxdb drop \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --confirm
```

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |
| `--confirm`            | Flag   | No       | Skip safety prompt                    |

### lxdb truncate[​](#lxdb-truncate "Direct link to lxdb truncate")

Clear all data from the LakeXpress DB, preserving the schema.

```
./LakeXpress lxdb truncate \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --confirm
```

| Option                 | Type   | Required | Description                            |
| ---------------------- | ------ | -------- | -------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                  |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file  |
| `--confirm`            | Flag   | No       | Skip safety prompt                     |
| `--sync_id ID`         | String | No       | Only truncate data for a specific sync |

### lxdb locks[​](#lxdb-locks "Direct link to lxdb locks")

Display locked tables in the LakeXpress DB.

note

Locks only apply to **incremental syncs**, protecting watermarks from concurrent modifications. Full syncs do not use locks.

```
./LakeXpress lxdb locks \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |
| `--sync_id ID`         | String | No       | Filter locks for a specific sync      |

### lxdb release-locks[​](#lxdb-release-locks "Direct link to lxdb release-locks")

Release stale or stuck locks.

**When to use:** After a sync crashes or is killed mid-export, stale locks block subsequent runs. Use `lxdb locks` to identify them, then `release-locks` to clear them.

```
# View current locks first
./LakeXpress lxdb locks \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres

# Release stale locks (requires --confirm)
./LakeXpress lxdb release-locks \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --confirm

# Release only locks older than 24 hours
./LakeXpress lxdb release-locks \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --max_age_hours 24 \
  --confirm

# Release a specific table lock by ID
./LakeXpress lxdb release-locks \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --table_id 42 \
  --confirm
```

| Option                 | Type    | Required | Description                           |
| ---------------------- | ------- | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path    | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String  | Yes      | LakeXpress DB identifier in auth file |
| `--confirm`            | Flag    | Yes      | Required to confirm the operation     |
| `--max_age_hours N`    | Integer | No       | Only release locks older than N hours |
| `--table_id ID`        | Integer | No       | Release lock for a specific table ID  |

## Configuration Management[​](#configuration-management "Direct link to Configuration Management")

### config create[​](#config-create "Direct link to config create")

Create a sync configuration stored in the LakeXpress DB.

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id source_pg \
  --source_db_name tpch \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --n_jobs 4 \
  --generate_metadata
```

#### Authentication Options[​](#authentication-options "Direct link to Authentication Options")

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |

#### Sync Identity Options[​](#sync-identity-options "Direct link to Sync Identity Options")

| Option         | Type   | Required | Description                                                                                                                                                                           |
| -------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--sync_id ID` | String | No       | Custom sync ID (1-64 chars, alphanumeric, underscores, hyphens)<br />Examples: `my_sync`, `prod-daily-export`, `sync_2026`<br />Auto-generates if omitted. Fails if ID already exists |

#### Source Database Options[​](#source-database-options "Direct link to Source Database Options")

| Option                         | Type   | Required | Description                                                             |
| ------------------------------ | ------ | -------- | ----------------------------------------------------------------------- |
| `--source_db_auth_id ID`       | String | Yes      | Source database identifier in auth file                                 |
| `--source_db_name NAME`        | String | Yes      | Source database name (e.g., `tpch`, `northwind`)                        |
| `--source_schema_name PATTERN` | String | Yes      | Source schema name(s), supports SQL patterns (e.g., `public`, `prod_%`) |

#### Table Filtering Options[​](#table-filtering-options "Direct link to Table Filtering Options")

| Option                  | Type    | Description                                                                               |
| ----------------------- | ------- | ----------------------------------------------------------------------------------------- |
| `-i, --include PATTERN` | String  | Include tables matching SQL patterns (comma-separated)<br />Example: `orders%, customer%` |
| `-e, --exclude PATTERN` | String  | Exclude tables matching SQL patterns (comma-separated)<br />Example: `temp%, test%`       |
| `--min_rows INT`        | Integer | Minimum row count filter                                                                  |
| `--max_rows INT`        | Integer | Maximum row count filter                                                                  |

**Pattern Matching:** Uses SQL `LIKE` syntax -- `%` matches any characters, `_` matches one character.

#### Incremental Sync Options[​](#incremental-sync-options "Direct link to Incremental Sync Options")

| Option                         | Type    | Description                                                                                                                                                                    |
| ------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--incremental_table SPEC`     | String  | Define an incremental table. Repeat the flag for each table.<br />Format: `schema.table:column:type[:i\|:e][@start][!strategy]`<br />Example: `tpch_1.orders:o_orderdate:date` |
| `--incremental_safety_lag INT` | Integer | Safety lag in seconds for late-arriving data (default: 0)                                                                                                                      |

note

Tables not configured with `--incremental_table` are fully exported on each sync.

**Incremental Column Types:**

* `date` - YYYY-MM-DD
* `datetime` - YYYY-MM-DD HH:MM
  <!-- -->
  :SS
* `timestamp` - Timestamp
* `integer` - Integer sequence

**Direction Options:**

* `:i` - Include (default)
* `:e` - Exclude

**Example:**

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id source_pg \
  --source_db_name tpch \
  --source_schema_name tpch_1_incremental \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
  --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
  --incremental_safety_lag 3600
```

See [Incremental Sync Documentation](/0.3/advanced/incremental-sync.md) for details.

#### FastBCP Configuration Options[​](#fastbcp-configuration-options "Direct link to FastBCP Configuration Options")

| Option                          | Type    | Default | Description                                                                                                                                                                                                                          |
| ------------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--fastbcp_dir_path PATH`       | Path    | N/A     | FastBCP executable directory                                                                                                                                                                                                         |
| `-p, --fastbcp_p INT`           | Integer | -4      | Parallel jobs within FastBCP for large table partitioning                                                                                                                                                                            |
| `--fastbcp_table_config CONFIG` | String  | N/A     | Table-specific FastBCP config<br />Format: `[schema.]table:method:key_column:p[;...]`<br />Example: `dbo.lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4`<br />Timepartition: `ORDERS:Timepartition:(ORDER_DATE,year,month):8` |
| `--large_table_threshold INT`   | Integer | 100000  | Row count threshold for parallel export                                                                                                                                                                                              |
| `--compression_type TYPE`       | String  | Zstd    | Parquet compression (Zstd, Snappy, Gzip, Lz4, None)                                                                                                                                                                                  |

#### Parallel Processing Options[​](#parallel-processing-options "Direct link to Parallel Processing Options")

| Option         | Type    | Default | Description                          |
| -------------- | ------- | ------- | ------------------------------------ |
| `--n_jobs INT` | Integer | 4       | Number of parallel table export jobs |

**Example:** `--n_jobs 4 --fastbcp_p 2` exports 4 tables simultaneously, each using 2 parallel processes.

#### Storage Options[​](#storage-options "Direct link to Storage Options")

Choose **either** `--output_dir` (local) **or** `--target_storage_id` (cloud).

| Option                   | Type   | Mutually Exclusive With | Description                                                                                                            |
| ------------------------ | ------ | ----------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--output_dir PATH`      | Path   | `--target_storage_id`   | Local directory for exports                                                                                            |
| `--target_storage_id ID` | String | `--output_dir`          | Cloud storage ID (e.g., `s3_01`, `gcs_01`, `azure_01`)                                                                 |
| `--sub_path SUB_PATH`    | String | N/A                     | Sub-path between base path and schema directory<br />Example: `staging/temp` creates `base/staging/temp/schema/table/` |

#### Publishing Options[​](#publishing-options "Direct link to Publishing Options")

| Option                             | Type   | Description                                                                                                                                                                                 |
| ---------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--publish_target ID`              | String | Credential ID for publishing target (Snowflake, AWS Glue, Databricks, Fabric, DuckLake)                                                                                                     |
| `--publish_method METHOD`          | String | `external` (default) -- data stays in cloud storage<br />`internal` -- data loaded into target database                                                                                     |
| `--publish_database_name NAME`     | String | Database name for publishing targets (AWS Glue, Databricks)                                                                                                                                 |
| `--publish_schema_pattern PATTERN` | String | Dynamic schema naming using tokens:<br />`{schema}`, `{table}`, `{database}`, `{date}`, `{timestamp}`, `{uuid}`, `{subpath}`<br />Default: `EXT_{schema}` (external), `{schema}` (internal) |
| `--publish_table_pattern PATTERN`  | String | Dynamic table naming (same tokens as schema pattern)<br />Default: `{table}`. Must include `{table}` token                                                                                  |
| `--no_views`                       | Flag   | Skip view creation (Snowflake external tables only)                                                                                                                                         |
| `--snowflake_pk_constraints`       | Flag   | Propagate PRIMARY KEY constraints to Snowflake internal tables                                                                                                                              |

#### Metadata Options[​](#metadata-options "Direct link to Metadata Options")

| Option                 | Type   | Default | Description                                                                               |
| ---------------------- | ------ | ------- | ----------------------------------------------------------------------------------------- |
| `--generate_metadata`  | Flag   | false   | Generate CDM metadata (`manifest.json` and `.cdm.json` files)                             |
| `--manifest_name NAME` | String | Auto    | Custom CDM manifest name<br />Default: schema name (per-schema) or database name (global) |

#### Behavior Options[​](#behavior-options "Direct link to Behavior Options")

| Option                  | Type   | Default | Description                                                              |
| ----------------------- | ------ | ------- | ------------------------------------------------------------------------ |
| `--error_action ACTION` | String | fail    | `fail` -- stop on first error<br />`continue` -- log errors and continue |
| `--env_name NAME`       | String | default | Environment name for configuration isolation                             |

#### Logging Options[​](#logging-options "Direct link to Logging Options")

| Option              | Type   | Default           | Description                                                                                                    |
| ------------------- | ------ | ----------------- | -------------------------------------------------------------------------------------------------------------- |
| `--log_level LEVEL` | String | INFO              | DEBUG, INFO, WARNING, ERROR, CRITICAL                                                                          |
| `--log_dir PATH`    | Path   | Current directory | Log file directory                                                                                             |
| `--no_progress`     | Flag   | false             | Disable progress bar                                                                                           |
| `--quiet_fbcp`      | Flag   | false             | Suppress FastBCP log lines from LakeXpress logs.<br />Statistics (rows, throughput, files) are still captured. |

### config list[​](#config-list "Direct link to config list")

List all sync configurations.

```
./LakeXpress config list \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |
| `--env_name NAME`      | String | No       | Filter by environment name            |

### config delete[​](#config-delete "Direct link to config delete")

Delete a sync configuration and all associated data (runs, table metadata, watermarks).

**Recommended workflow:** Run without `--confirm` first to preview what will be deleted, then run with `--confirm` to execute.

```
# Step 1: Dry run - preview what will be deleted
./LakeXpress config delete \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id 20251208-a1b2c3d4-e5f6-7890

# Step 2: Confirm deletion
./LakeXpress config delete \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id 20251208-a1b2c3d4-e5f6-7890 \
  --confirm
```

| Option                 | Type   | Required | Description                                                  |
| ---------------------- | ------ | -------- | ------------------------------------------------------------ |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                                        |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file                        |
| `--sync_id ID`         | String | Yes      | Sync configuration ID to delete                              |
| `--confirm`            | Flag   | No       | Execute the deletion (without this flag, shows preview only) |

**What gets deleted:**

* The sync configuration
* All run history for this sync
* Table metadata and watermarks (incremental sync state)

note

This does not delete exported files in cloud storage or published tables in target systems (Snowflake, Glue, etc.).

## Sync Execution[​](#sync-execution "Direct link to Sync Execution")

### sync[​](#sync "Direct link to sync")

Execute a sync configuration.

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id ID \
    --sync_id <SYNC_ID>
```

| Option                    | Type   | Description                                                          |
| ------------------------- | ------ | -------------------------------------------------------------------- |
| `-a, --auth_file PATH`    | Path   | **Required.** Path to credentials JSON file                          |
| `--lxdb_auth_id ID`       | String | **Required.** Credential ID for the log database                     |
| `--sync_id ID`            | String | **Required.** Sync configuration to execute                          |
| `--fastbcp_dir_path PATH` | Path   | Override FastBCP directory                                           |
| `--resume`                | Flag   | Resume from last incomplete run                                      |
| `--run_id ID`             | String | Specific run ID to resume<br />Format: `YYYYMMDD-XXXXXXXX-XXXX-XXXX` |

**Example:**

```
# Execute a sync
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id my-sync-id

# Resume incomplete run
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id my-sync-id \
    --run_id 20251208-f7g8h9i0-j1k2-l3m4 \
    --resume
```

### sync\[export][​](#syncexport "Direct link to sync\[export]")

Export data without publishing. Same required options as `sync`.

```
./LakeXpress 'sync[export]' \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id my-sync-id
```

### sync\[publish][​](#syncpublish "Direct link to sync\[publish]")

Publish previously exported data to Snowflake, AWS Glue, Databricks, Fabric, BigQuery, MotherDuck, or DuckLake. Same required options as `sync`.

```
./LakeXpress 'sync[publish]' \
    -a credentials.json \
    --lxdb_auth_id lxdb \
    --sync_id my-sync-id
```

## Legacy YAML Support[​](#legacy-yaml-support "Direct link to Legacy YAML Support")

### run[​](#run "Direct link to run")

Execute an export from a legacy YAML configuration file.

```
./LakeXpress run \
  -c config_20251202_164948.yml \
  -a credentials.json
```

| Option                 | Type   | Required | Description                          |
| ---------------------- | ------ | -------- | ------------------------------------ |
| `-c, --config PATH`    | Path   | Yes      | YAML configuration file              |
| `-a, --auth_file PATH` | Path   | No       | Override credentials file            |
| `--lxdb_auth_id ID`    | String | No       | Override LakeXpress DB credential ID |

note

YAML files are auto-generated by `config create` but superseded by database-stored configurations.

## Status and Monitoring[​](#status-and-monitoring "Direct link to Status and Monitoring")

### status[​](#status "Direct link to status")

Query sync and run status.

```
./LakeXpress status \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id 20251208-a1b2c3d4-e5f6-7890
```

| Option                 | Type   | Required | Description                           |
| ---------------------- | ------ | -------- | ------------------------------------- |
| `-a, --auth_file PATH` | Path   | Yes      | JSON credentials file                 |
| `--lxdb_auth_id ID`    | String | Yes      | LakeXpress DB identifier in auth file |
| `--sync_id ID`         | String | No       | Filter by sync configuration          |
| `--run_id ID`          | String | No       | Filter by run                         |
| `-v, --verbose`        | Flag   | No       | Show detailed run list                |

## Cleanup and Maintenance[​](#cleanup-and-maintenance "Direct link to Cleanup and Maintenance")

### cleanup[​](#cleanup "Direct link to cleanup")

Remove orphaned or stale runs from the LakeXpress DB.

```
./LakeXpress cleanup \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id my_sync \
  --older-than 7d \
  --dry-run
```

| Option                  | Type   | Required | Description                                                              |
| ----------------------- | ------ | -------- | ------------------------------------------------------------------------ |
| `-a, --auth_file PATH`  | Path   | Yes      | JSON credentials file                                                    |
| `--lxdb_auth_id ID`     | String | Yes      | LakeXpress DB identifier in auth file                                    |
| `--sync_id ID`          | String | Yes      | Sync configuration to clean up                                           |
| `--older-than DURATION` | String | No       | Only delete runs older than this (e.g., `7d`, `24h`, `30m`)              |
| `--status STATUS`       | String | No       | Only delete runs with this status: `running` or `failed` (default: both) |
| `--dry-run`             | Flag   | No       | Preview deletions without executing                                      |

## Complete Examples[​](#complete-examples "Direct link to Complete Examples")

### Basic Export to S3[​](#basic-export-to-s3 "Direct link to Basic Export to S3")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name sales_db \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --n_jobs 4 \
  --fastbcp_p 2

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

### Export with Snowflake Publishing[​](#export-with-snowflake-publishing "Direct link to Export with Snowflake Publishing")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name sales_db \
  --source_schema_name public \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --publish_target snowflake_prod \
  --publish_schema_pattern "EXT_{subpath}_{date}" \
  --publish_table_pattern "{schema}_{table}" \
  --sub_path production \
  --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

### Incremental Export[​](#incremental-export "Direct link to Incremental Export")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_ms \
  --source_db_auth_id ds_04_pg \
  --source_db_name tpch \
  --source_schema_name tpch_1_incremental \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id aws_s3_01 \
  --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
  --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
  --generate_metadata

# First sync -- exports all data
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --sync_id <SYNC_ID>

# Subsequent syncs -- incremental tables export only new data, others fully exported
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_ms \
    --sync_id <SYNC_ID>
```

### Export with Custom Naming and Table Filtering[​](#export-with-custom-naming-and-table-filtering "Direct link to Export with Custom Naming and Table Filtering")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name analytics \
  --source_schema_name "sales%, marketing%" \
  --include "fact_%, dim_%" \
  --exclude "temp%, test%" \
  --fastbcp_dir_path ./FastBCP_linux-x64/latest/ \
  --target_storage_id s3_01 \
  --sub_path data-lake/prod \
  --publish_target snowflake_prod \
  --publish_schema_pattern "ANALYTICS_{subpath}" \
  --publish_table_pattern "{schema}_{table}" \
  --n_jobs 8 \
  --fastbcp_p 4 \
  --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/0.3/guides/quickstart.md)
* [Incremental Sync](/0.3/advanced/incremental-sync.md)
* [Intermediate Storage](/0.3/configuration/storage/storage-overview.md)
* [Database Configuration](/0.3/configuration/database-configuration.md)
* [Examples & Recipes](/0.3/guides/command-cookbook.md)


---

# Logging and Monitoring

LakeXpress uses two logging systems: a database for structured export tracking and log files for operational details.

## Database Logging[​](#database-logging "Direct link to Database Logging")

The export tracking database (SQL Server, PostgreSQL, or SQLite) records all export activity for reporting and troubleshooting.

### Database Schema Overview[​](#database-schema-overview "Direct link to Database Schema Overview")

### Key Tables[​](#key-tables "Direct link to Key Tables")

#### export\_runs[​](#export_runs "Direct link to export_runs")

Top-level record per export run:

* Unique run identifier
* Run type (full, incremental, custom)
* Source database and schema
* Storage configuration
* Start/completion timestamps and status
* Aggregate metrics (total, completed, failed jobs)
* Total rows exported and storage size

#### export\_jobs[​](#export_jobs "Direct link to export_jobs")

Per-table export job tracking:

* Job ID and associated run\_id
* Step ID linking to export\_steps
* Source table details (database, schema, table)
* Storage settings (type, path, format, compression)
* Parquet settings (row group size)
* FastBCP settings (distribution key, method, parallel degree)
* Results (file count, rows exported)
* Status (job\_status, step\_status)
* Retry count and timestamps (created, started, completed)

#### export\_steps[​](#export_steps "Direct link to export_steps")

Workflow step definitions:

* Step names and descriptions
* Links to jobs and log entries

#### export\_files[​](#export_files "Direct link to export_files")

Per-file tracking for generated Parquet files:

* File ID linked to parent job
* File name and full path
* Storage location (local, S3, Azure, GCS)
* File size, row count, status, creation timestamp

#### export\_columns[​](#export_columns "Direct link to export_columns")

Column metadata per exported table:

* Job ID + source column name (composite PK)
* Source database type
* Parquet column name (post-transformation)
* Source data type, precision, scale
* Ordinal position, length, nullability

#### export\_log[​](#export_log "Direct link to export_log")

Event log:

* Log ID and timestamp
* Links to run\_id, job\_id, step\_id
* Log level (INFO, WARNING, ERROR, DEBUG)
* Event status, elapsed time (seconds)
* Source table context (database, schema, table)
* Event/error messages and error codes

#### partition\_columns[​](#partition_columns "Direct link to partition_columns")

FastBCP partitioning configuration:

* Partition config ID
* Source table identification (db type, database, schema, table)
* FastBCP distribution key, method, parallel degree
* Environment name, enable/disable flag
* Description and audit timestamps

## File-Based Logging[​](#file-based-logging "Direct link to File-Based Logging")

Each run creates a log file named: `lakexpress_YYYYMMDD_HHMMSS_<run_id>.log`

### Log Format[​](#log-format "Direct link to Log Format")

Structured format with timestamp, log level, process ID, and context:

```
[YYYY-MM-DD HH:MM:SS.fff+TZ :: LEVEL :: PID :: Context] Message
```

### Sample Log Output[​](#sample-log-output "Direct link to Sample Log Output")

```
[2025-11-05 10:15:23.145+01:00 :: INFO :: 12345 :: MainProcess] Log file path: /var/log/lakexpress/lakexpress_20251105_101523_a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d.log
[2025-11-05 10:15:23.146+01:00 :: INFO :: 12345 :: MainProcess] **** Starting export run ****
[2025-11-05 10:15:23.147+01:00 :: INFO :: 12345 :: MainProcess] Source: SQLServer - salesdb.dbo
[2025-11-05 10:15:23.148+01:00 :: INFO :: 12345 :: MainProcess] Target storage: S3 - s3://data-lake/exports/
[2025-11-05 10:15:23.250+01:00 :: INFO :: 12345 :: MainProcess] Discovered 15 tables for export
[2025-11-05 10:15:23.385+01:00 :: INFO :: 12345 :: Worker-1] Exporting table: orders (estimated 1.2M rows)
[2025-11-05 10:15:45.721+01:00 :: INFO :: 12345 :: Worker-1] Export completed: orders - 1,234,567 rows in 8 files (245 MB)
[2025-11-05 10:15:45.722+01:00 :: INFO :: 12345 :: Worker-1] Export elapsed time: 22.337 seconds
```

### Log Levels[​](#log-levels "Direct link to Log Levels")

Set via `--log-level`:

INFO

Progress, row counts, operational messages

WARNING

Non-critical issues (schema differences, type conversions)

ERROR

Failures preventing a table export

DEBUG

SQL queries, file operations, diagnostics

CRITICAL

Fatal errors stopping the run

### Log Location[​](#log-location "Direct link to Log Location")

Defaults to the current working directory. Override with `--log-dir`:

```
./lakexpress --log-dir /var/log/lakexpress export ...
```

### Terminal Output[​](#terminal-output "Direct link to Terminal Output")

Logs also appear in the terminal, color-coded:

* Green: success
* Yellow: warnings
* Red: errors
* Cyan: info


---

# Sitemap

All pages of this documentation, in the order of the sidebar.

* [Overview](/0.3/.md)
* [Compatibility](/0.3/compatibility.md)
* [Installation](/0.3/installation.md)
* [Getting Started](/0.3/guides/quickstart.md)
* [Command Wizard](/0.3/wizard.md)
* [Examples & Recipes](/0.3/guides/command-cookbook.md)

## Data Pipeline Connections

* [Database Connections](/0.3/configuration/database-configuration.md)

* LakeXpress DB

  * [PostgreSQL](/0.3/configuration/databases/postgresql.md)
  * [SQL Server](/0.3/configuration/databases/sql-server.md)
  * [MySQL](/0.3/configuration/databases/mysql.md)
  * [SQLite](/0.3/configuration/databases/sqlite.md)
  * [DuckDB](/0.3/configuration/databases/duckdb.md)

* Source Databases

  * [PostgreSQL](/0.3/configuration/databases/postgresql.md)
  * [Oracle](/0.3/configuration/databases/oracle.md)
  * [SQL Server](/0.3/configuration/databases/sql-server.md)
  * [MySQL](/0.3/configuration/databases/mysql.md)
  * [MariaDB](/0.3/configuration/databases/mariadb.md)
  * [SAP HANA](/0.3/configuration/databases/sap-hana.md)
  * [Teradata](/0.3/configuration/databases/teradata.md)

* Intermediate Storage

  * [Storage Overview](/0.3/configuration/storage/storage-overview.md)
  * [Local Filesystem](/0.3/configuration/storage/local-filesystem.md)
  * [S3-Compatible Storage](/0.3/configuration/storage/s3.md)
  * [Google Cloud Storage (GCS)](/0.3/configuration/storage/gcs.md)
  * [Azure Storage](/0.3/configuration/storage/azure.md)
  * [Microsoft Fabric OneLake](/0.3/configuration/storage/onelake.md)

* Target Platform

  * Snowflake

    * [Snowflake Publishing](/0.3/integrations/snowflake-publishing.md)
    * [Snowflake Reference](/0.3/integrations/snowflake-reference.md)

  * Databricks

    * [Databricks Publishing](/0.3/integrations/databricks-publishing.md)
    * [Databricks Reference](/0.3/integrations/databricks-reference.md)

  * AWS Glue

    * [AWS Glue Publishing](/0.3/integrations/glue-publishing.md)
    * [AWS Glue Reference](/0.3/integrations/glue-reference.md)

  * Microsoft Fabric

    * [Microsoft Fabric Publishing](/0.3/integrations/fabric-publishing.md)
    * [Microsoft Fabric Reference](/0.3/integrations/fabric-reference.md)

  * BigQuery

    * [BigQuery Publishing](/0.3/integrations/bigquery-publishing.md)
    * [BigQuery Reference](/0.3/integrations/bigquery-reference.md)

  * Amazon Redshift
    * [Amazon Redshift Publishing](/0.3/integrations/redshift-publishing.md)

  * MotherDuck

    * [MotherDuck Publishing](/0.3/integrations/motherduck-publishing.md)
    * [MotherDuck Reference](/0.3/integrations/motherduck-reference.md)

  * [ducklake-publishing](/0.3/integrations/ducklake-publishing.md)

## Advanced

* [Export Performance](/0.3/advanced/export-performance.md)

* Incremental Sync

  * [Incremental Sync](/0.3/advanced/incremental-sync.md)
  * [Loading Strategies](/0.3/advanced/incremental/loading-strategies.md)
  * [Watermarks](/0.3/advanced/incremental/watermarks.md)
  * [Examples](/0.3/advanced/incremental/examples.md)
  * [Troubleshooting](/0.3/advanced/incremental/troubleshooting.md)

## Reference

* [CLI Reference](/0.3/reference/cli-reference.md)
* [Logging and Monitoring](/0.3/reference/logging.md)


---

# Command Wizard

Build your LakeXpress command step by step using the interactive wizard below.

<!-- -->

1. 1Set UpWindows · SQL Server
2. 2SourceSQL Server
3. 3StorageLocal
4. 4PublishOff
5. 5TuneDefaults
6. 6OptionsDefaults

Step

<!-- -->

1

<!-- -->

of

<!-- -->

6

### Set Up

#### Platform?

Windows

![Linux](/img/tux.svg)Linux

#### Credentials

| Authentication File? : | credentials.json                                                  |
| ---------------------- | ----------------------------------------------------------------- |
| Sync ID (optional)? :  | Auto-generated if empty1-64 chars, alphanumeric/underscore/hyphen |

#### LakeXpress DB

SQL Server

PostgreSQL

MySQL

SQLite

DuckDB

| LakeXpress DB ID? : | e.g., lxdb\_sqlserver |
| ------------------- | --------------------- |

Next:

<!-- -->

Source

### Generated Commands

ShareMulti-line\[x]

1. 1

   Initialize the LakeXpress DB

   **Run once:** creates the LakeXpress tables. Safe to run again.

   ```
   .\LakeXpress.exe lxdb init `
     -a credentials.json `
     --lxdb_auth_id <lxdb_auth_id>
   ```

2. 2

   Create the sync configuration

   **Run once:** saves the configuration in the LakeXpress DB and prints the generated sync ID.

   ```
   .\LakeXpress.exe config create `
     -a credentials.json
   ```

3. 3

   Run the sync

   Exports the tables.

   ```
   .\LakeXpress.exe sync `
     -a credentials.json `
     --lxdb_auth_id <lxdb_auth_id> `
     --sync_id <sync_id printed by config create>
   ```


---

# LakeXpress

## What is LakeXpress?[​](#what-is-lakexpress "Direct link to What is LakeXpress?")

LakeXpress is a CLI tool that exports database tables to partitioned Parquet files and publishes them to cloud data platforms. It uses [FastBCP](https://www.arpe.io/fastbcp/?v=82a9e4d26595) to stream data in parallel without exhausting memory.

## Architecture[​](#architecture "Direct link to Architecture")

LakeXpress connects four components in a data pipeline:

### Source Database[​](#source-database "Direct link to Source Database")

The database you want to export data from. LakeXpress reads tables (not views) from the source and converts them to Parquet files.

Supported sources

Connection setup and required privileges for each source database

* [![](/img/logos/oracle.svg)Oracle](/latest/configuration/databases/oracle.md)
* [![](/img/logos/postgresql.svg)PostgreSQL](/latest/configuration/databases/postgresql.md)
* [![](/img/logos/sqlserver.svg)SQL Server](/latest/configuration/databases/sql-server.md)
* [![](/img/logos/mysql.svg)MySQL](/latest/configuration/databases/mysql.md)
* [![](/img/logos/mariadb.svg)MariaDB](/latest/configuration/databases/mariadb.md)
* [![](/img/logos/sap.svg)SAP HANA](/latest/configuration/databases/sap-hana.md)
* [![](/img/logos/teradata.svg)Teradata](/latest/configuration/databases/teradata.md)

### Intermediate Storage[​](#intermediate-storage "Direct link to Intermediate Storage")

Parquet files are written to a storage backend, where they can be used as they are or published to a target platform. This can be local disk or cloud object storage.

Supported storage backends

Credentials and paths for each storage backend. S3-compatible storage (MinIO, OVH...) uses the AWS S3 page

* [![](/img/logos/local-files.svg)Local filesystem](/latest/configuration/storage/local-filesystem.md)
* [![](/img/logos/aws-s3.png)AWS S3](/latest/configuration/storage/s3.md)
* [![](/img/logos/gcs.png)Google Cloud Storage](/latest/configuration/storage/gcs.md)
* [![](/img/logos/azure-blob.svg)Azure Blob Storage](/latest/configuration/storage/azure.md)
* [![](/img/logos/onelake.png)Microsoft Fabric OneLake](/latest/configuration/storage/onelake.md)

### Target Platform[​](#target-platform "Direct link to Target Platform")

Optional. Once Parquet files are in storage, LakeXpress can publish them to a cloud data platform or catalog, either as **external** tables (the data stays in storage) or as **internal** tables (the data is loaded into the platform). See [Compatibility](/latest/reference/compatibility.md#supported-publishing-targets) for the methods each platform supports.

Supported platforms

Setup, authentication and options for each platform

* [![](/img/logos/snowflake.svg)Snowflake](/latest/integrations/snowflake-publishing.md)
* [![](/img/logos/databricks.svg)Databricks](/latest/integrations/databricks-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric](/latest/integrations/fabric-publishing.md)
* [Amazon Redshift](/latest/integrations/redshift-publishing.md)
* [![](/img/logos/bigquery.svg)BigQuery](/latest/integrations/bigquery-publishing.md)
* [![](/img/logos/motherduck.svg)MotherDuck](/latest/integrations/motherduck-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue](/latest/integrations/glue-publishing.md)
* [![](/img/logos/ducklake.svg)DuckLake](/latest/integrations/ducklake-publishing.md)

### LakeXpress DB[​](#lakexpress-db "Direct link to LakeXpress DB")

A dedicated database where LakeXpress stores its sync configurations, the history and status of every run, and the watermarks of incremental syncs. Every LakeXpress command uses it, so it must be set up before the first export. It also enables tracking runs and resuming failed exports.

See [Set up the LakeXpress DB](/latest/lakexpress-db/.md) for an overview.

Supported LakeXpress DBs

How to create and initialize the LakeXpress DB on each database

* [![](/img/logos/postgresql.svg)PostgreSQL](/latest/lakexpress-db/postgresql.md)
* [![](/img/logos/sqlserver.svg)SQL Server](/latest/lakexpress-db/sql-server.md)
* [![](/img/logos/mysql.svg)MySQL](/latest/lakexpress-db/mysql.md)
* [SQLite](/latest/lakexpress-db/sqlite.md)
* [![](/img/logos/duckdb.svg)DuckDB](/latest/lakexpress-db/duckdb.md)

## What you'll need[​](#what-youll-need "Direct link to What you'll need")

| Component                          | What to do                                                                                                          | Details                                                                                                                                                                          |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **LakeXpress binary**              | Download and unzip                                                                                                  | [Installation guide](/latest/installation.md)                                                                                                                                    |
| **Source database user**           | Create a read-only user with SELECT on target schemas                                                               | [Database setup](/latest/configuration/database-configuration.md)                                                                                                                |
| **LakeXpress DB**                  | An empty PostgreSQL, SQL Server or MySQL database owned by its user, or a SQLite or DuckDB file (nothing to create) | [Set up the LakeXpress DB](/latest/lakexpress-db/.md)                                                                                                                            |
| **Storage destination**            | A local directory or cloud storage credentials                                                                      | [Storage config](/latest/configuration/storage/storage-overview.md)                                                                                                              |
| **Publishing target** *(optional)* | Credentials for Snowflake, Databricks, etc.                                                                         | [Snowflake](/latest/integrations/snowflake-publishing.md) · [Databricks](/latest/integrations/databricks-publishing.md) · [more...](/latest/integrations/bigquery-publishing.md) |

## Key features[​](#key-features "Direct link to Key features")

* Cross-platformNative binaries for Windows and Linux
* Parallel exportsMultiple tables at once, with per-table partitioning
* Incremental syncWatermark-based delta exports
* Schema filteringInclude/exclude schemas and tables via SQL patterns
* Resume on failurePick up where a failed export left off
* CDM metadataGenerate Common Data Model files

## How it works[​](#how-it-works "Direct link to How it works")

1. 1Initialize`LakeXpress lxdb init` creates the LakeXpress tables in the LakeXpress DB (once)
2. 2Configure`LakeXpress config create` defines your source database, storage target, and optional publishing
3. 3Sync`LakeXpress sync` exports tables to Parquet and, if a publishing target is configured, publishes them

See the [Quick Start Guide](/latest/guides/quickstart.md) for a full walkthrough with real commands.


---

# Export Performance

LakeXpress auto-detects the best export method per table. Override when needed using CLI flags or the `partition_columns` table in the LakeXpress DB.

## Export Methods[​](#export-methods "Direct link to Export Methods")

### Without Key Column[​](#without-key-column "Direct link to Without Key Column")

`None`All databases

Sequential export, no parallelism

`Ctid`![](/img/logos/postgresql.svg)PostgreSQL

Tuple ID-based parallel export

`Rowid`![](/img/logos/oracle.svg)Oracle

Row ID-based parallel export

`Physloc`![](/img/logos/sqlserver.svg)SQL Server

Physical location-based parallel export

### With Key Column[​](#with-key-column "Direct link to With Key Column")

`Random`

Distributes using column values

Key columnInteger or bigint

`DataDriven`

Splits on distinct values

Key columnColumn with distinct values, e.g. year, region

`RangeId`

Range partitioning via min/max

Key columnNumeric or date column

`Ntile`

Even distribution

Key columnNumeric column

[`Timepartition`](#timepartition)

Partitions by a date/time column into Hive-style paths

Key columnDate or datetime column, e.g. (col,year,month)

### Timepartition[​](#timepartition "Direct link to Timepartition")

Splits data into time-based chunks where each parallel thread handles a specific time period. Works with any source database -- ideal for time-series data and large historical tables.

**Key column format** -- a parenthesized list of a date column followed by granularity levels:

`(col, year)`year=YYYY

`(col, year, month)`year=YYYY/month=MM

`(col, year, month, day)`year=YYYY/month=MM/day=DD

`(col, year, week)`year=YYYY/week=WW

Exported files follow Hive-style partitioned paths, e.g. `year=2008/month=01/CARRIER_CLAIMS.parquet`. Rows whose date is NULL go to a `<col>=NULL/` directory, e.g. `CLM_FROM_DT=NULL/CARRIER_CLAIMS.parquet`.

## Tuning Parameters[​](#tuning-parameters "Direct link to Tuning Parameters")

`--n_jobs`default `1`

Tables exported simultaneously

`--fastbcp_p`or`-p`default `1`

Parallel threads per large table

`--large_table_threshold`default `100000`

Row count below which tables export sequentially

`--compression_type`default `Zstd`

Parquet compression: `Zstd`, `Snappy`, `Gzip`, `Lz4` or `None`

Both `--n_jobs` and `--fastbcp_p` default to `1` (sequential) — increase them explicitly to enable parallelism. `--fastbcp_p` only takes effect when its value is greater than `1`; tables below `--large_table_threshold` use the `None` (sequential) method regardless of `--fastbcp_p`, unless a per-table method is set (see [Per-Table CLI Override](#per-table-cli-override) and [Per-Table Database Override](#per-table-database-override)): a per-table method applies whatever the row count.

**Parallel degree guidelines** (`--fastbcp_p`, threads per table):

1

< 100K

<!-- -->

rows

Default. Below the large-table threshold, parallel export is skipped

2–4

100K–10M

<!-- -->

rows

4–8

10M–100M

<!-- -->

rows

8–16

\> 100M

<!-- -->

rows

### Incremental Tables[​](#incremental-tables "Direct link to Incremental Tables")

An incremental table (`--incremental_table`) is exported with a query that filters on its watermark column, for the first sync as for the later ones. The methods without a key column (`Ctid`, `Rowid`, `Physloc`) need a whole table and cannot be used with a query: LakeXpress replaces them with `None` and logs a warning, whether it chose the method itself or you set it with `--fastbcp_table_config`. A large PostgreSQL, Oracle or SQL Server incremental table is therefore exported with a single thread by default.

To keep parallelism, set a method with a key column for that table:

```
--incremental_table "fact.transactions:txn_date:date" \
--fastbcp_table_config "fact.transactions:Ntile:txn_id:8"
```

## Per-Table CLI Override[​](#per-table-cli-override "Direct link to Per-Table CLI Override")

```
--fastbcp_table_config "[schema.]table:method:key_column:parallel_degree"
```

The schema prefix is optional. When omitted, the config applies to all schemas containing that table (a warning is logged). Separate several tables with semicolons, or repeat `--fastbcp_table_config`.

The parallel degree is **required** and must be an integer: there is no default. An entry without a valid degree is ignored, without an error. The method and key column can be left empty (e.g. `orders:::4`) to keep the auto-detected method with the given degree. A database-specific method used with another database (e.g. `Ctid` on SQL Server) is replaced by the right one for that database, with a warning.

```
# Schema-qualified (targets specific schema)
--fastbcp_table_config "dbo.lineitem:DataDriven:YEAR(l_shipdate):8;dbo.orders:Ctid::4"

# Without schema (applies to all schemas, with warning)
--fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4"

# Timepartition -- key column uses parenthesized format
--fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4"
```

## Per-Table Database Override[​](#per-table-database-override "Direct link to Per-Table Database Override")

Insert into the `partition_columns` table for persistent configuration:

```
INSERT INTO partition_columns (
    source_db_type, source_database, source_schema, source_table,
    fastbcp_method, fastbcp_parallel_degree, fastbcp_distribution_key
) VALUES (
    'postgres', 'analytics', 'public', 'events',
    'Ctid', 4, NULL
);
```

A row only applies to syncs created with the same `--env_name` (column `env_name`, default `'default'`) and while `is_enabled` is true (the default). Set `env_name` in the `INSERT` if your sync uses another environment name.

Database configuration takes priority over CLI configuration, which takes priority over auto-detection.

## Monitoring[​](#monitoring "Direct link to Monitoring")

The real table is `export_jobs` (there is no plain `jobs` table), with columns `job_status`, `started_at`/`completed_at`, and `num_rows_exported`:

```
SELECT source_table, job_status, started_at, completed_at, num_rows_exported
FROM export_jobs
WHERE run_id = 'your-run-id'
ORDER BY started_at DESC;
```


---

# Incremental Sync

Sync only changed data using watermark-based incremental exports.

## What is Incremental Sync?[​](#what-is-incremental-sync "Direct link to What is Incremental Sync?")

LakeXpress tracks a "high watermark" (the highest value in a timestamp or numeric column) and only exports rows above that watermark on subsequent syncs, instead of exporting entire tables every time.

## When to Use Incremental Sync[​](#when-to-use-incremental-sync "Direct link to When to Use Incremental Sync")

Use incremental sync when:

* Regularly updated tablesSource tables receive frequent inserts or updates
* Frequent syncsExports run multiple times per day or on a schedule
* Cost efficiencyYou want to minimize network and compute usage
* Time-series dataTables have timestamp columns tracking record creation or modification
* Monotonic identifiersTables have an auto-incrementing or sequential key (identity, serial, or sequence-backed ID) that reliably increases with each insert, usable as an `int`/`bigint` watermark even when no timestamp column exists

**Examples:**

* Daily sales order updates to a data lake
* Event log aggregation from production systems
* Time-series metrics collection
* Transaction processing pipelines
* Append-only tables keyed by an auto-increment ID (no modified-at column)

## Supported Column Types[​](#supported-column-types "Direct link to Supported Column Types")

| Type       | Format                                | Example             | Use Case                                                                                      |
| ---------- | ------------------------------------- | ------------------- | --------------------------------------------------------------------------------------------- |
| `date`     | YYYY-MM-DD                            | 2025-01-08          | Daily transactions, order dates                                                               |
| `datetime` | YYYY-MM-DD HH:MM<!-- -->:SS           | 2025-01-08 14:30:25 | Precise time tracking, creation/modification timestamps                                       |
| `int`      | Numeric sequence                      | 1000001             | Monotonic ID columns, batch IDs                                                               |
| `bigint`   | Large numeric sequence                | 9000000001          | High-volume monotonic ID columns                                                              |
| `auto`     | Numeric (treated like `int`/`bigint`) | —                   | Let LakeXpress determine the watermark via `MAX(column)` without specifying `int` vs `bigint` |

caution

These five tokens are the only ones accepted. `timestamp` and `integer` are **not** valid — passing either raises `IncrementalTableParseError` (`sync_config_parser.py`, `VALID_INCREMENTAL_TYPES = {"date", "datetime", "int", "bigint", "auto"}`). Use `datetime` instead of `timestamp`, and `int`/`bigint` instead of `integer`.

## Configuration Syntax[​](#configuration-syntax "Direct link to Configuration Syntax")

Define incremental tables with `--incremental_table`:

```
--incremental_table "schema.table:column:type"
```

### Basic Syntax[​](#basic-syntax "Direct link to Basic Syntax")

`schema.table`TableFully qualified table name:`column`Watermark columnColumn to track, preferably indexed:`type`Column typedate, datetime, int, bigint or auto

**Example:**

```
--incremental_table "sales.orders:created_date:date"
```

### Multiple Tables[​](#multiple-tables "Direct link to Multiple Tables")

Repeat the `--incremental_table` flag for each table:

```
--incremental_table "sales.orders:created_date:date"
--incremental_table "sales.returns:return_date:date"
```

### Advanced Syntax (Optional)[​](#advanced-syntax-optional "Direct link to Advanced Syntax (Optional)")

`schema.table`Table:`column`Watermark column:`type`Column type`:i | :e`Boundoptional`@start_value`Start valueoptional`!strategy`Strategyoptional

`:i`or`:e`default `:e`

The watermark **comparison bound**, not an include/exclude toggle for the table. `:i` is an inclusive lower bound (`WHERE column >= watermark`), `:e` an exclusive one (`WHERE column > watermark`).

`@start_value`

Start the first (baseline) export at this value: the baseline exports `column >= start_value AND column <= d_new`, always inclusive whatever `:i`/`:e` says. Later runs start from the stored watermark.

`!strategy`default `append`

Loading strategy: `append`, `upsert` or `delete_insert`. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

caution

`:i`/`:e` only changes the `>=` vs `>` comparison used against the watermark column — every table listed under `--incremental_table` is always synced incrementally. There is no per-table "exclude from incremental sync" option; a table is either listed with `--incremental_table` (incremental) or left out (fully re-exported every run).

**Examples:**

```
# Inclusive lower bound: WHERE created_date >= watermark
--incremental_table "sales.orders:created_date:date:i"

# Exclusive lower bound (default, shown explicitly): WHERE return_date > watermark
--incremental_table "sales.returns:return_date:date:e"

# Set initial watermark
--incremental_table "sales.orders:created_date:date@2025-01-01"

# Use UPSERT/MERGE strategy (updates existing rows)
--incremental_table "sales.orders:created_date:date!upsert"

# Combined: inclusive bound + start value + upsert
--incremental_table "sales.orders:created_date:date:i@2025-01-01!upsert"
```

## Non-incremental Tables[​](#non-incremental-tables "Direct link to Non-incremental Tables")

Tables not configured with `--incremental_table` are fully exported on each sync. Useful for small dimension or reference tables.

```
--incremental_table "fact.sales:sale_date:date"
--n_jobs 4
```

* `fact.sales`Exports only the new records since the last watermark
* All other tables in the schemaFully exported on each sync

## See Also[​](#see-also "Direct link to See Also")

* [Loading Strategies](/latest/advanced/incremental/loading-strategies.md)Append vs upsert strategies
* [Watermarks](/latest/advanced/incremental/watermarks.md)Watermark tracking, safety lag, querying, and resetting
* [Incremental Sync recipes](/latest/guides/recipes/incremental-sync.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/latest/advanced/incremental/troubleshooting.md)Common issues, performance tips, and advanced topics
* [Quick Start Guide](/latest/guides/quickstart.md)First export walkthrough
* [CLI Reference](/latest/cli/.md)All available options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Real-world usage examples
* [Intermediate Storage](/latest/configuration/storage/storage-overview.md)Cloud storage configuration


---

# Loading Strategies

The strategy is set per table, at the end of `--incremental_table`: `!append`, `!upsert` or `!delete_insert`. It decides how the new rows of each sync are written to the target platform.

| Strategy           | What it does                                                                            | Primary key |
| ------------------ | --------------------------------------------------------------------------------------- | ----------- |
| `append` (default) | Adds the new rows                                                                       | Not needed  |
| `upsert`           | Updates the rows that already exist (matched on the primary key) and inserts the others | Required    |
| `delete_insert`    | Deletes the rows whose primary key is in the new batch, then inserts the batch          | Required    |

## Support by Target Platform[​](#support-by-target-platform "Direct link to Support by Target Platform")

| Target                       | `append`  | `upsert`      | `delete_insert` | Table used for `upsert` / `delete_insert`           |
| ---------------------------- | --------- | ------------- | --------------- | --------------------------------------------------- |
| Snowflake                    | Supported | Supported     | Supported       | Internal table, created automatically               |
| Amazon Redshift              | Supported | Supported     | Supported       | Internal table, created automatically               |
| Databricks                   | Supported | Supported     | Supported       | Managed table: requires `--publish_method internal` |
| BigQuery                     | Supported | Supported     | Supported       | Native table: requires `--publish_method internal`  |
| Microsoft Fabric (Warehouse) | Supported | Supported     | Supported       | Warehouse tables are always internal                |
| MotherDuck                   | Supported | Supported     | Supported       | Internal table, created automatically               |
| DuckLake                     | Supported | Supported     | Supported       | DuckLake table, created automatically               |
| Microsoft Fabric (Lakehouse) | Supported | Not supported | Not supported   | —                                                   |
| AWS Glue                     | Supported | Not supported | Not supported   | —                                                   |

On Snowflake, Redshift, MotherDuck and DuckLake, the tables that use `upsert` or `delete_insert` are always created as persistent internal tables, whatever `--publish_method` is. The other tables of the same sync keep the table type set by `--publish_method`.

MotherDuck and DuckLake have no `MERGE` statement: on these targets, `upsert` is performed as a delete followed by an insert, which gives the same result for the whole-row Parquet exports LakeXpress produces.

Fabric uses the **Warehouse** target unless the credentials entry sets `"fabric_target": "lakehouse"` (see [Microsoft Fabric](/latest/integrations/fabric-publishing.md)).

LakeXpress checks every table before publishing. If a strategy is not supported by the target, or a table using `upsert` or `delete_insert` has no primary key, the publish step stops with an error that names the table.

## Append Strategy (Default)[​](#append-strategy-default "Direct link to Append Strategy (Default)")

Inserts new rows into the target table. Suitable when:

* Your target handles duplicates (e.g., external tables)
* You're building an append-only data lake
* You dedup at query time

```
--incremental_table "events.pageviews:event_time:datetime!append"
```

## Upsert Strategy[​](#upsert-strategy "Direct link to Upsert Strategy")

Updates the existing rows and inserts the new ones, so the target always reflects the current state of the source, without duplicates.

```
--incremental_table "sales.orders:updated_at:datetime!upsert"
```

**How it works:**

1. The new rows are loaded into a staging table
2. `MERGE` runs: matching rows are updated, new rows are inserted (on MotherDuck and DuckLake: matching rows are deleted, then all new rows are inserted)
3. The staging table is dropped

**Requirements:**

* The source table must have a primary key. LakeXpress detects it during schema discovery.
* The target must support it (see the table above). On Databricks and BigQuery, use `--publish_method internal`.

**Example with BigQuery:**

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name ecommerce \
  --source_schema_name public \
  --target_storage_id gcs_01 \
  --incremental_table "public.orders:updated_at:datetime!upsert" \
  --incremental_table "public.customers:updated_at:datetime!upsert" \
  --publish_target bigquery_prod \
  --publish_method internal \
  --n_jobs 4
```

**Example with Snowflake** (the upsert tables are created as internal tables, even with the default `external` publish method):

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --source_db_auth_id source_mssql \
  --source_db_name sales \
  --source_schema_name dbo \
  --target_storage_id s3_01 \
  --incremental_table "dbo.products:modified_date:datetime!upsert" \
  --publish_target snowflake_prod \
  --n_jobs 4
```

## Delete-Insert Strategy[​](#delete-insert-strategy "Direct link to Delete-Insert Strategy")

Deletes the rows of the target whose primary key appears in the new batch, then inserts the whole batch. Each changed row is replaced as a whole.

```
--incremental_table "sales.orders:updated_at:datetime!delete_insert"
```

Same requirements as `upsert`: a primary key on the source table, and a target that supports it.

## Choosing a Strategy[​](#choosing-a-strategy "Direct link to Choosing a Strategy")

| Scenario                                | Recommended Strategy |
| --------------------------------------- | -------------------- |
| Append-only event logs                  | `append`             |
| Transaction history                     | `append`             |
| Time-series metrics                     | `append`             |
| CDC (Change Data Capture)               | `upsert`             |
| Customer/product master data            | `upsert`             |
| Order status updates                    | `upsert`             |
| Rows rewritten as a whole by the source | `delete_insert`      |

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/latest/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Incremental Sync recipes](/latest/guides/recipes/incremental-sync.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/latest/advanced/incremental/troubleshooting.md)Handling duplicates and other common issues


---

# Troubleshooting

## Missing Late-Arriving Rows[​](#missing-late-arriving-rows "Direct link to Missing Late-Arriving Rows")

**Symptom:** Rows inserted in the source after a sync are never exported.

**Cause:** Each run exports up to `MAX(column)` read at run start and stores it as the watermark. A row committed later with a value at or below that watermark is outside every later run's range.

**Solution:** For `datetime` and `date` columns, set `--incremental_safety_lag` to the longest delay between a row's timestamp and its commit, so the most recent rows wait for the next run (see [Safety Lag](/latest/advanced/incremental/watermarks.md#safety-lag)). Otherwise use an incremental column whose values only grow as rows are written (identity column, or a load timestamp set by the database), or reload the table periodically (see [Resetting Watermarks](/latest/advanced/incremental/watermarks.md#resetting-watermarks)).

## Duplicate Rows After Resync[​](#duplicate-rows-after-resync "Direct link to Duplicate Rows After Resync")

**Symptom:** Running the same sync twice produces duplicates.

**Solutions:**

1. **Use the upsert strategy** (recommended; not available on AWS Glue and Fabric Lakehouse, see [Loading Strategies](/latest/advanced/incremental/loading-strategies.md#support-by-target-platform)): configure the table with `!upsert` so existing rows are updated instead of added again. The source table needs a primary key:

```
--incremental_table "sales.orders:created_date:date!upsert"
```

2. **Deduplicate in queries**: If using append mode, use ROW\_NUMBER() or QUALIFY at query time.

3. **Use external tables**: External tables over Parquet files replace files on each sync rather than inserting rows.

## Watermark Not Advancing[​](#watermark-not-advancing "Direct link to Watermark Not Advancing")

**Symptom:** Watermark stays at the same value across syncs.

**Solution:** The watermark is `MAX(column)` read from the source at the start of each run, for every column type. If it does not change, no row with a higher value has arrived: check with the query below. If the source has newer values, check on the [Watermarks](/latest/advanced/incremental/watermarks.md#querying-watermarks) page that `last_exported_value` in `incremental_watermarks` is updated (a sync failing before the end of the export does not update it).

```
SELECT MAX(created_at) FROM orders;
-- If this equals the stored watermark, there is no new data to export.
```

## Performance Considerations[​](#performance-considerations "Direct link to Performance Considerations")

### Column Selection[​](#column-selection "Direct link to Column Selection")

Choose the incremental column carefully:

```
-- GOOD: Indexed column, never updates
--incremental_table "sales.orders:order_id:int"

-- GOOD: Updated on insert only
--incremental_table "sales.orders:created_date:date"

-- CAUTION: May need resync if updated
--incremental_table "sales.orders:updated_at:datetime"

-- AVOID: May miss updates
--incremental_table "sales.orders:order_amount:int"
```

### Indexing[​](#indexing "Direct link to Indexing")

Create an index on the watermark column:

```
CREATE INDEX idx_orders_created_at ON orders(created_at);
```

### Combining Incremental and Full Exports[​](#combining-incremental-and-full-exports "Direct link to Combining Incremental and Full Exports")

```
--incremental_table "fact.transactions:txn_date:date"
--incremental_safety_lag 3600
--n_jobs 4
--fastbcp_p 8
--fastbcp_table_config "fact.transactions:Ntile:txn_id:8"
```

* Fact tables configured with `--incremental_table` sync incrementally
* All other tables in the schema are fully exported (useful for dimension tables)
* `--fastbcp_p` alone does not parallelize an incremental table on PostgreSQL, Oracle or SQL Server: the automatic methods (`Ctid`, `Rowid`, `Physloc`) cannot be used with the incremental query and are replaced by `None`. The `--fastbcp_table_config` entry sets a method with a key column (`Ntile` on `txn_id`) for it. See [Incremental Tables](/latest/advanced/export-performance.md#incremental-tables).

## Multiple Watermark Columns[​](#multiple-watermark-columns "Direct link to Multiple Watermark Columns")

Track changes in multiple columns with separate configurations:

```
# Configuration 1: Track by create date
lakexpress config create \
  ... \
  --incremental_table "sales.transactions:created_at:datetime"

# Configuration 2: Track by update date
lakexpress config create \
  ... \
  --incremental_table "sales.transactions:updated_at:datetime"
```

Run both syncs to capture creates and updates separately.

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/latest/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Loading Strategies](/latest/advanced/incremental/loading-strategies.md)Append vs upsert strategies
* [Watermarks](/latest/advanced/incremental/watermarks.md)Watermark tracking, safety lag, and resetting


---

# Watermarks

## Watermark Tracking[​](#watermark-tracking "Direct link to Watermark Tracking")

LakeXpress maintains watermarks in the LakeXpress DB. Every export — baseline or incremental — has a lower bound `d_old` (the previous watermark, `NULL` on the first run) and an upper bound `d_new`:

* `d_new` is `MAX(column)`, read from the source table at the start of the run, for every column type (`datetime`, `date`, `int`, `bigint`, `auto`).
* For `datetime` and `date` columns with `--incremental_safety_lag`, `d_new` is capped at the source's current time (or date) minus the lag: `d_new = min(MAX(column), now - lag)`. See [Safety Lag](#safety-lag).
* `d_new` never goes below the stored watermark: if the lag puts it lower (for example after raising the lag), `d_new` is the watermark, the run exports no rows for that table, and the watermark stays where it is.
* If the table is empty (`MAX(column)` is `NULL`), `d_new` is the source's current time (or date) minus `--incremental_safety_lag` for `datetime`/`date` columns.

The generated `WHERE` clause is:

* Baseline (first run): `WHERE column <= d_new`, or `WHERE column >= start_value AND column <= d_new` when the table is configured with `@start_value` (always inclusive, whatever `:i`/`:e` says)
* Incremental: `WHERE column > d_old AND column <= d_new` (or `>=` if the table was configured with `:i`)

### First Sync (Baseline)[​](#first-sync-baseline "Direct link to First Sync (Baseline)")

```
Table: orders
Column: o_orderdate (datetime)
d_old: NULL
d_new: 2025-12-31 22:47:12  (= MAX(o_orderdate) at run start)
Query: WHERE o_orderdate <= '2025-12-31 22:47:12'
Records exported: 1,500,000
Watermark stored: last_exported_value = 2025-12-31 22:47:12
```

### Second Sync (Incremental)[​](#second-sync-incremental "Direct link to Second Sync (Incremental)")

```
d_old: 2025-12-31 22:47:12  (previous watermark)
d_new: 2026-01-05 07:58:30  (= MAX(o_orderdate) at run start)
Query: WHERE o_orderdate > '2025-12-31 22:47:12' AND o_orderdate <= '2026-01-05 07:58:30'
Expected records: 50,000 (new orders since the last run)
Watermark updated: last_exported_value = 2026-01-05 07:58:30
```

## Late-Arriving Data[​](#late-arriving-data "Direct link to Late-Arriving Data")

Without a safety lag, `d_new` is the largest value present in the source when the run starts, so a row committed later with a value lower than or equal to the stored watermark is not exported by the following runs. This happens with long transactions, or when the application sets the timestamp before the commit.

For `datetime` and `date` columns, set `--incremental_safety_lag` to the longest delay between a row's timestamp and its commit: rows newer than `now - lag` are left for the next run. For other columns, choose an incremental column whose values only grow as rows are written (an identity column, or a load timestamp set by the database), or periodically reload the table (see [Resetting Watermarks](#resetting-watermarks)).

## Safety Lag[​](#safety-lag "Direct link to Safety Lag")

`--incremental_safety_lag INT` (seconds, default: 0) holds back the most recent rows of `datetime` and `date` columns:

* `d_new = min(MAX(column), now - lag)`. A row whose value is later than `now - lag` is exported by a later run, once it is older than the lag.
* `now` is read from the source database (`CURRENT_TIMESTAMP`), because the column holds the source's clock, usually without a time zone. If it cannot be read, LakeXpress uses UTC and logs a warning.
* For `date` columns, the cutoff is a date: a lag of `86400` (one day) exports up to yesterday.
* `int`, `bigint` and `auto` columns are not affected.

```
MAX(updated_at):        2026-10-29 11:59:30
Source time:            2026-10-29 12:00:00
--incremental_safety_lag 300
d_new:                  2026-10-29 11:55:00
```

## Querying Watermarks[​](#querying-watermarks "Direct link to Querying Watermarks")

Inspect tracked watermarks by querying the LakeXpress DB directly. There is no `sync_configurations` table — incremental state lives across three real tables: `incremental_tables` (one row per configured table), `incremental_watermarks` (current state, one row per `incremental_tables` row), and `incremental_export_history` (one row per run):

```
-- View current watermark state for every incremental table in a sync
SELECT
    t.sync_id,
    t.source_schema,
    t.source_table,
    t.incremental_column,
    t.incremental_type,
    t.lower_bound_inclusive,   -- true = :i (>=), false = :e (>, default)
    t.strategy,
    w.last_exported_value,     -- d_old for the next run
    w.d_new,                   -- upper bound used by the last run
    w.last_export_at,
    w.rows_exported
FROM incremental_tables t
LEFT JOIN incremental_watermarks w ON w.incremental_table_id = t.incremental_table_id
WHERE t.sync_id = 'YOUR_SYNC_ID'
ORDER BY t.source_schema, t.source_table;

-- View recent export history (across all watermarks/tables)
SELECT
    h.run_id,
    t.source_schema,
    t.source_table,
    h.d_old,
    h.d_new,
    h.export_mode,       -- 'baseline' or 'incremental'
    h.rows_exported,
    h.status,            -- 'completed', 'failed', 'partial', 'empty'
    h.started_at,
    h.completed_at
FROM incremental_export_history h
INNER JOIN incremental_watermarks w ON w.watermark_id = h.watermark_id
INNER JOIN incremental_tables t ON t.incremental_table_id = w.incremental_table_id
ORDER BY h.completed_at DESC
LIMIT 10;
```

## Resetting Watermarks[​](#resetting-watermarks "Direct link to Resetting Watermarks")

There is no `--reset-watermarks` CLI flag — no such option exists in LakeXpress. To force a full reload:

```
# Option 1 (supported): delete and recreate the configuration.
# --confirm is required — without it, `config delete` only prints a dry-run
# summary of what would be deleted and does not touch the database.
./LakeXpress config delete \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id 20251208-xxxxx \
  --confirm

# Then create a new one with the same options: it gets a new sync_id and fresh watermarks
```

```
-- Option 2 (manual, not a first-class feature): clear the watermark of a single
-- table directly in the LakeXpress DB. With last_exported_value = NULL, the next
-- sync runs that table as a new baseline. Do not DELETE the row: once the table
-- has run, its export history references it (foreign key).
UPDATE incremental_watermarks
SET last_exported_value = NULL, d_new = NULL
WHERE incremental_table_id = (
    SELECT incremental_table_id
    FROM incremental_tables
    WHERE sync_id = 'YOUR_SYNC_ID'
      AND source_schema = 'sales'
      AND source_table = 'orders'
);
```

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync Overview](/latest/advanced/incremental-sync.md)Configuration syntax and supported column types
* [Incremental Sync recipes](/latest/guides/recipes/incremental-sync.md)Complete step-by-step examples and real-world scenarios
* [Troubleshooting](/latest/advanced/incremental/troubleshooting.md)Fixing watermark issues and missing data


---

# LakeXpress Agent Skill

arpe.io publishes official agent skills in the [arpe-io/arpeio-skills](https://github.com/arpe-io/arpeio-skills) GitHub repository. The [`lakexpress`](https://github.com/arpe-io/arpeio-skills/tree/main/plugins/lakexpress/skills/lakexpress) skill teaches your AI agent how to build, explain and troubleshoot LakeXpress pipelines: the credentials file, sync configurations, incremental syncs, publishing to a lakehouse, and resuming a failed export.

The same repository also has skills for FastBCP, FastTransfer, MigratorXpress and the arpe.io ADBC drivers.

Skill or MCP server?

The skill is a set of instructions: it teaches the agent how LakeXpress works, and the agent writes the command. The [arpe.io MCP server](/latest/ai/mcp-server.md) gives the agent tools that build, validate and, if you allow it, run LakeXpress commands. The two work well together, so install both if your agent supports them.

## Claude Code[​](#claude-code "Direct link to Claude Code")

Install the skill as a plugin from the arpe.io marketplace:

```
/plugin install lakexpress --marketplace arpe-io/arpeio-skills
```

Or in two steps:

```
/plugin marketplace add arpe-io/arpeio-skills
/plugin install lakexpress@arpe-io
```

Then just ask, for example: *"Export the sales schema from PostgreSQL to Parquet on S3 and publish it to Snowflake with LakeXpress"*.

To get the latest version of the skill later:

```
/plugin marketplace update arpe-io
/plugin update lakexpress@arpe-io
```

## Other agents (Codex, Cursor, GitHub Copilot, Gemini CLI…)[​](#other-agents-codex-cursor-github-copilot-gemini-cli "Direct link to Other agents (Codex, Cursor, GitHub Copilot, Gemini CLI…)")

Copy the skill folder into the directory your agent reads skills from:

```
git clone https://github.com/arpe-io/arpeio-skills.git
cp -r arpeio-skills/plugins/lakexpress/skills/lakexpress <your-agent-skills-dir>/
```

For agents without skill support, paste the content of [`SKILL.md`](https://github.com/arpe-io/arpeio-skills/blob/main/plugins/lakexpress/skills/lakexpress/SKILL.md) as custom instructions. The files in its `references/` folder can be attached or pasted when needed.

## Product version[​](#product-version "Direct link to Product version")

The skill states the LakeXpress version its facts were checked against. Before building a command, it asks the agent to check the installed version (`LakeXpress --version`) and to confirm against this documentation when the installed product is newer.

## Reporting issues[​](#reporting-issues "Direct link to Reporting issues")

If the agent gets something wrong, open an issue at [github.com/arpe-io/arpeio-skills/issues](https://github.com/arpe-io/arpeio-skills/issues) with the LakeXpress version, the prompt you gave, and the command the agent produced (with passwords and licence text removed).


---

# LakeXpress MCP Server

The [arpe.io MCP server](https://github.com/arpe-io/arpeio-mcp) (`arpeio-mcp`) gives AI assistants a set of tools for the arpe.io products: FastBCP, FastTransfer, MigratorXpress, LakeXpress and the arpe.io ADBC drivers. [MCP](https://modelcontextprotocol.io) (Model Context Protocol) is the open standard that Claude, ChatGPT, Cursor, VS Code and most other assistants use to call external tools.

For LakeXpress, the assistant can ask the server to validate a set of parameters and render the exact LakeXpress command, look up what LakeXpress supports, read the release notes and search this documentation. If LakeXpress is installed on your machine, it can also run the command.

The server runs in one of two modes:

* **Command builder**, the default. The tools build, validate and explain commands; no LakeXpress binary is needed. The hosted server works this way, so it never connects to your databases.
* **Execution**. When you install the server on your own machine and point it at your LakeXpress binary, the assistant can also run the command it built, once you confirm it.

MCP server or agent skill?

The MCP server gives the assistant tools to call. The [agent skill](/latest/ai/agent-skills.md) is a set of instructions that teaches it how the product works. The two work well together, so install both if your assistant supports them.

## Connect your assistant[​](#connect-your-assistant "Direct link to Connect your assistant")

arpe.io hosts the server, so there is nothing to install. Add this URL to your assistant:

```
https://arpe-io-arpeio-mcp.hf.space/sse
```

* Claude Code
* Claude Desktop
* ChatGPT
* Cursor
* VS Code (Copilot)
* Gemini CLI
* Windsurf
* Le Chat (Mistral)

```
claude mcp add --transport sse arpeio https://arpe-io-arpeio-mcp.hf.space/sse
```

Add the server to your Claude Desktop configuration file:

* **Linux**: `~/.config/Claude/claude_desktop_config.json`
* **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

```
{
  "mcpServers": {
    "arpeio": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://arpe-io-arpeio-mcp.hf.space/sse"
      ]
    }
  }
}
```

Custom connectors need a paid plan (Plus, Pro, Team or Enterprise).

1. Open ChatGPT in your browser, go to **Settings** > **Apps and connectors**.
2. Open **Advanced settings** and enable **Developer mode**.
3. Go to **Connectors** > **Browse connectors** > **Add a new connector**.
4. Set the URL to `https://arpe-io-arpeio-mcp.hf.space/sse` and save.

Open Cursor Settings, search for "MCP" and add a server with this configuration:

```
{
  "mcpServers": {
    "arpeio": {
      "url": "https://arpe-io-arpeio-mcp.hf.space/sse"
    }
  }
}
```

Run **MCP: Open User Configuration** from the Command Palette and add:

```
{
  "servers": {
    "arpeio": {
      "url": "https://arpe-io-arpeio-mcp.hf.space/sse",
      "type": "sse"
    }
  }
}
```

Add the server to `~/.gemini/settings.json`:

```
{
  "mcpServers": {
    "arpeio": {
      "uri": "https://arpe-io-arpeio-mcp.hf.space/sse"
    }
  }
}
```

Add the server to `~/.codeium/windsurf/mcp_config.json` (Windows: `%USERPROFILE%\.codeium\windsurf\mcp_config.json`):

```
{
  "mcpServers": {
    "arpeio": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://arpe-io-arpeio-mcp.hf.space/sse"
      ]
    }
  }
}
```

Available on all plans, including the free one.

1. Go to **Intelligence** > **Connectors**.
2. Click **Add connector** > **Custom MCP Connector**.
3. Set the name to `Arpe.io` and the URL to `https://arpe-io-arpeio-mcp.hf.space/sse`.
4. Leave authentication disabled and click **Create**.

Instructions for HuggingChat and Kiro are in the [server README](https://github.com/arpe-io/arpeio-mcp#connect-your-ai-assistant).

## Run LakeXpress from the assistant[​](#run-lakexpress-from-the-assistant "Direct link to Run LakeXpress from the assistant")

To let the assistant run LakeXpress, install the server on the machine where LakeXpress is installed (Python 3.10 or later):

```
pip install arpeio-mcp
```

Then register the `arpeio-mcp` command instead of the hosted URL, with the path to your binary. In Claude Code:

```
claude mcp add --transport stdio --env LAKEXPRESS_PATH=/path/to/LakeXpress --env FASTBCP_DIR_PATH=/path/to/FastBCP/directory arpeio -- arpeio-mcp
```

In Claude Desktop, and in other clients that use the `mcpServers` format:

```
{
  "mcpServers": {
    "arpeio": {
      "command": "arpeio-mcp",
      "env": {
        "LAKEXPRESS_PATH": "/path/to/LakeXpress",
        "FASTBCP_DIR_PATH": "/path/to/FastBCP/directory"
      }
    }
  }
}
```

| Variable             | Description                                                                 |
| -------------------- | --------------------------------------------------------------------------- |
| `LAKEXPRESS_PATH`    | Path to the LakeXpress binary. Without it, the server only builds commands. |
| `FASTBCP_DIR_PATH`   | Directory holding the FastBCP binary that LakeXpress uses for the exports.  |
| `LAKEXPRESS_TIMEOUT` | Execution timeout, in seconds. Optional.                                    |
| `LAKEXPRESS_LOG_DIR` | Directory for the execution logs. Optional.                                 |
| `LOG_LEVEL`          | Server log level: `DEBUG`, `INFO` (default), `WARNING` or `ERROR`.          |

## Tools[​](#tools "Direct link to Tools")

| Tool                         | What it does                                                                                                                        |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `lakexpress_info`            | Reference information, picked with `action`: capabilities, workflow tips, version detection.                                        |
| `lakexpress_preview_command` | Builds any LakeXpress command (`lxdb_*`, `config_*`, `sync`, `run`, `status`, `cleanup`…) and warns about missing required options. |
| `lakexpress_execute_command` | Runs the previewed command. Needs a local LakeXpress binary and `confirmation: true`.                                               |

Tools shared by all arpe.io products:

| Tool                 | What it does                                                                                            |
| -------------------- | ------------------------------------------------------------------------------------------------------- |
| `arpe_quick_start`   | Picks the right arpe.io product for a use case described in plain English and returns a workflow guide. |
| `arpe_get_status`    | Shows which arpe.io binaries the server can run, and which are in command-builder mode only.            |
| `arpe_release_notes` | Returns release notes for a product, or for one version.                                                |
| `search_docs`        | Full-text search over the arpe.io documentation sites, this one included, and the arpe.io blog.         |

## How a request flows[​](#how-a-request-flows "Direct link to How a request flows")

1. You describe what you want, for example: *"Set up a LakeXpress sync that exports the sales schema from SQL Server to Parquet on S3 and publishes it to Snowflake"*.
2. The assistant calls `lakexpress_preview_command`. The tool checks the parameters and returns the LakeXpress command, an explanation, and warnings when an option does not exist in the LakeXpress version you run.
3. You review the command. Passwords are masked in the displayed version.
4. If execution is enabled, the assistant calls `lakexpress_execute_command`. The tool refuses to run unless the call carries `confirmation: true`, and it returns the exit code, the output and a diagnosis of known errors.

Most clients also ask for your approval before each tool call.

## Prompts and resources[​](#prompts-and-resources "Direct link to Prompts and resources")

Clients that support MCP prompts (Claude Desktop, Cursor and others) list conversation starters: `lakehouse-pipeline` for LakeXpress, and `troubleshoot` to explain a failed run from its error message. The server also exposes the LakeXpress capability matrix as the MCP resource `arpeio://capabilities/lakexpress-capabilities`.

## Reporting issues[​](#reporting-issues "Direct link to Reporting issues")

If a tool returns a wrong command or wrong information, open an issue at [github.com/arpe-io/arpeio-mcp/issues](https://github.com/arpe-io/arpeio-mcp/issues) with the prompt you gave and the tool output (with passwords and licence text removed).


---

# CLI Reference

LakeXpress works in two steps:

1. **Create a sync configuration** with `config create`. It is stored in the LakeXpress DB.
2. **Run it** with `sync`, as often as needed.

The other commands manage the LakeXpress DB (`lxdb`), show run status (`status`) and remove stale runs (`cleanup`).

This section describes the syntax and options of each command. For complete, ready-to-run commands, see [Examples & Recipes](/latest/guides/command-cookbook.md).

## Commands[​](#commands "Direct link to Commands")

| Command                                                                          | Purpose                         |
| -------------------------------------------------------------------------------- | ------------------------------- |
| [`lxdb init`, `drop`, `truncate`, `locks`, `release-locks`](/latest/cli/lxdb.md) | Manage the LakeXpress DB        |
| [`config create`, `list`, `delete`](/latest/cli/config.md)                       | Manage sync configurations      |
| [`sync`, `sync[export]`, `sync[publish]`](/latest/cli/sync.md)                   | Run a sync configuration        |
| [`status`](/latest/cli/status.md)                                                | Show syncs and runs             |
| [`cleanup`](/latest/cli/cleanup.md)                                              | Remove stale or failed runs     |
| [`run`](/latest/cli/run.md)                                                      | Run a legacy YAML configuration |

## Command Line Structure[​](#command-line-structure "Direct link to Command Line Structure")

```
usage: LakeXpress [-h] [-v | --version] [--no_banner]
                  COMMAND [SUBCOMMAND] [OPTIONS]
```

`-h`, `--version` and `--no_banner` can be placed before the command. All other options go **after** the command, for example `LakeXpress sync --log_level DEBUG ...`.

## Common Options[​](#common-options "Direct link to Common Options")

Every command accepts these options. The pages of each command list only its own options.

| Option                 | Description                                                                                                                                                            |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a, --auth_file PATH` | Credentials JSON file. Required by every command except `run`                                                                                                          |
| `--lxdb_auth_id ID`    | Entry of the credentials file for the LakeXpress DB. Required by every command except `run`                                                                            |
| `--log_level LEVEL`    | `DEBUG`, `INFO` (default), `WARNING` or `ERROR`                                                                                                                        |
| `--log_dir PATH`       | Directory for the run log files, created if needed (default: current directory). Used by `sync`, `sync[export]` and `run`. See [Logging](/latest/reference/logging.md) |
| `--no_progress`        | Disable the progress bars (export and publish)                                                                                                                         |
| `--no_banner`          | Do not print the startup banner                                                                                                                                        |
| `--quiet_fbcp`         | Do not copy FastBCP log lines into the LakeXpress logs (statistics are still recorded)                                                                                 |
| `--license TEXT`       | License text (alternative to the `LAKEXPRESS_LICENSE` environment variable)                                                                                            |
| `--license_file PATH`  | Path to a license file                                                                                                                                                 |
| `-h, --help`           | Show the help of the command and exit                                                                                                                                  |

`LakeXpress --version` (or `-v`) shows the version and the date the binary was built.

## License Resolution[​](#license-resolution "Direct link to License Resolution")

LakeXpress looks for a license in this order:

1. `LAKEXPRESS_LICENSE` environment variable

2. `--license`

3. `--license_file`

4. Default file locations:

   <!-- -->

   * **Linux:** `~/.lakexpress/license.lic`
   * **Windows:** `%APPDATA%/LakeXpress/license.lic`
   * **Current directory:** `./license.lic`

There is no license-free trial: the free 30-day trial is a license file from [arpe.io](https://arpe.io/get-trial?product=LakeXpress). Without a valid license, every command except `--help` and `--version` exits with code 13 (`LICENSE_ERROR`).

The license file also carries the FastBCP and ADBC driver licenses: keep it whole, as delivered.

### Examples[​](#examples "Direct link to Examples")

**A license file outside the default locations.** Add `--license_file` to every command, after the command name:

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID> \
  --license_file /opt/lakexpress/license/LakeXpress.lic
```

```
.\LakeXpress.exe sync `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID> `
  --license_file C:\licenses\LakeXpress.lic
```

**Scheduled jobs (cron, Task Scheduler, Airflow).** Use an absolute path: the job's working directory and home directory are often not the ones of your interactive session, so the default locations may not be found.

```
# crontab: daily sync at 02:00
0 2 * * * cd /opt/lakexpress && ./LakeXpress sync -a /opt/lakexpress/credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID> --license_file /opt/lakexpress/license/LakeXpress.lic --no_banner --no_progress
```

**CI and containers: the `LAKEXPRESS_LICENSE` environment variable.** It holds the license text itself, for example from a CI secret, so no file has to be copied onto the machine:

```
export LAKEXPRESS_LICENSE="$(cat /opt/lakexpress/license/LakeXpress.lic)"
./LakeXpress sync -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID>
```

```
$env:LAKEXPRESS_LICENSE = Get-Content -Raw C:\licenses\LakeXpress.lic
.\LakeXpress.exe sync -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID>
```

In a container, either set `LAKEXPRESS_LICENSE` from your secret store, or mount the file read-only and point to it:

```
docker run --rm \
  -v /opt/lakexpress/license/LakeXpress.lic:/license/LakeXpress.lic:ro \
  <your-lakexpress-image> \
  sync -a /config/credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID> \
  --license_file /license/LakeXpress.lic
```

warning

* `LAKEXPRESS_LICENSE` comes first in the lookup order: when it is set, `--license` and `--license_file` are ignored, even if the variable holds an invalid license (exit 13). Unset it (`unset LAKEXPRESS_LICENSE`, `Remove-Item Env:LAKEXPRESS_LICENSE`) if a command does not pick up the file you pass.
* Prefer `--license_file` or `LAKEXPRESS_LICENSE` to `--license "<text>"`: a command-line argument is visible to other users of the machine in the process list (`ps`, Task Manager).


---

# cleanup

Remove runs of a sync configuration that are stuck in `running` (for example after a crash) or `failed`, from the LakeXpress DB. Completed runs are never removed. Exported files are not deleted.

```
usage: LakeXpress cleanup --auth_file AUTH_FILE --lxdb_auth_id ID
                          --sync_id SYNC_ID [--older-than DURATION]
                          [--status {running,failed}] [--dry-run]
                          [common options]
```

| Option                  | Description                                                              |
| ----------------------- | ------------------------------------------------------------------------ |
| `--sync_id ID`          | **Required.** Sync configuration to clean up                             |
| `--older-than DURATION` | Only remove runs older than this, e.g. `7d`, `24h`, `30m`, `60s`         |
| `--status STATUS`       | Only remove runs with this status: `running` or `failed` (default: both) |
| `--dry-run`             | Show what would be removed, without removing anything                    |

Also accepts the [common options](/latest/cli/.md#common-options).


---

# config

Manage sync configurations. A sync configuration says what to export, where, and where to publish it. It is stored in the LakeXpress DB and run with [`sync`](/latest/cli/sync.md). Examples: [Configuration Management recipes](/latest/guides/recipes/configuration-management.md), and the other [Examples & Recipes](/latest/guides/command-cookbook.md).

All `config` commands also accept the [common options](/latest/cli/.md#common-options).

## config create[​](#config-create "Direct link to config create")

Create a sync configuration. The command prints its `sync_id` and the `sync` command to run it.

```
usage: LakeXpress config create --auth_file AUTH_FILE
                                --lxdb_auth_id ID
                                --source_db_auth_id ID [OPTIONS]
                                [common options]
```

At least one of `--output_dir` (local disk) or `--target_storage_id` (cloud storage) is required. `config create` does not check it: a configuration without either fails when the sync runs.

### Sync Identity[​](#sync-identity "Direct link to Sync Identity")

| Option            | Description                                                                                                                                          |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--sync_id ID`    | Name of the sync configuration: 1 to 64 letters, digits, `_` or `-` (e.g. `prod-daily-export`). Generated if omitted. Fails if the ID already exists |
| `--env_name NAME` | Environment name, to keep separate configurations for dev, test and prod (default: `default`)                                                        |

### Source[​](#source "Direct link to Source")

| Option                         | Description                                                                                                     |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| `--source_db_auth_id ID`       | **Required.** Entry of the credentials file for the source database                                             |
| `--source_db_name NAME`        | Source database name. Default: `info.database` of the source credentials entry                                  |
| `--source_schema_name PATTERN` | Schemas to export, comma-separated, SQL `LIKE` patterns allowed (e.g. `public`, `prod_%`). Default: all schemas |

### Table Filtering[​](#table-filtering "Direct link to Table Filtering")

| Option                  | Description                                                                                 |
| ----------------------- | ------------------------------------------------------------------------------------------- |
| `-i, --include PATTERN` | Only export the tables matching these patterns, comma-separated (e.g. `orders%, customer%`) |
| `-e, --exclude PATTERN` | Skip the tables matching these patterns, comma-separated (e.g. `temp%, test%`)              |
| `--min_rows N`          | Skip tables with fewer than N rows                                                          |
| `--max_rows N`          | Skip tables with more than N rows                                                           |

Patterns use SQL `LIKE` syntax: `%` matches any characters, `_` matches one character.

### Incremental Sync[​](#incremental-sync "Direct link to Incremental Sync")

| Option                             | Description                                                                                                                                                                                                                                                                            |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--incremental_table SPEC`         | Export only new rows of a table, tracked with a watermark column. Repeat for each table. Format: `schema.table:column:type[:i\|:e][@start][!strategy]`, e.g. `tpch_1.orders:o_orderdate:date`                                                                                          |
| `--incremental_safety_lag SECONDS` | For `datetime` and `date` columns, cap the upper bound at the source's current time minus this many seconds: `min(MAX(column), now - lag)`, so late-committed rows are exported by the next run (default: 0). See [Watermarks](/latest/advanced/incremental/watermarks.md#safety-lag). |

Column types: `date`, `datetime` (also for timestamp columns), `int`, `bigint`, `auto`. Lower bound: `:e` exclusive, `column > watermark` (default), or `:i` inclusive, `column >= watermark`. Tables without `--incremental_table` are fully exported on each sync. Details: [Incremental Sync](/latest/advanced/incremental-sync.md).

### Output[​](#output "Direct link to Output")

| Option                   | Description                                                                                                                                                                         |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--output_dir PATH`      | Local directory for the Parquet files. A relative path is resolved against the current directory and saved as an absolute path (0.4.9+; use an absolute path with earlier versions) |
| `--target_storage_id ID` | Entry of the credentials file for the cloud storage (S3, GCS, Azure, OneLake)                                                                                                       |
| `--sub_path PATH`        | Folder added between the base path and the schema folders, e.g. `staging/temp` gives `base/staging/temp/schema/table/`                                                              |

### Export Engine[​](#export-engine "Direct link to Export Engine")

| Option                          | Description                                                                                                                                                                                                                                                                                  |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--fastbcp_dir_path PATH`       | Folder containing FastBCP. Not needed: LakeXpress finds the FastBCP bundled in its `engine/` folder                                                                                                                                                                                          |
| `--n_jobs N`                    | Number of tables exported in parallel (default: 1)                                                                                                                                                                                                                                           |
| `-p, --fastbcp_p N`             | Parallel processes used by FastBCP within one large table (default: 1)                                                                                                                                                                                                                       |
| `--large_table_threshold N`     | Row count above which a table is exported in parallel (default: 100000)                                                                                                                                                                                                                      |
| `--fastbcp_table_config CONFIG` | Per-table FastBCP settings: `[schema.]table:method:key_column:degree`, separated by `;`, or repeat the option. The degree is required: an entry without it is ignored. E.g. `dbo.lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4`, or `ORDERS:Timepartition:(ORDER_DATE,year,month):8` |
| `--compression_type TYPE`       | Parquet compression: `Zstd` (default), `Snappy`, `Gzip`, `Lz4` or `None`                                                                                                                                                                                                                     |

With `--n_jobs 4 --fastbcp_p 2`, four tables are exported at the same time, each with two processes.

### Publishing[​](#publishing "Direct link to Publishing")

| Option                             | Description                                                                                                                                                                                                                                                                    |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--publish_target ID`              | Entry of the credentials file for the target platform (Snowflake, Databricks, AWS Glue, Fabric, BigQuery, Redshift, MotherDuck, DuckLake)                                                                                                                                      |
| `--publish_method METHOD`          | `external` (default): tables read the Parquet files in cloud storage. `internal`: data is loaded into the platform. Fabric Lakehouse defaults to `internal`                                                                                                                    |
| `--publish_database_name NAME`     | AWS Glue: Glue database name. BigQuery: dataset name. Replaces `--publish_schema_pattern` when set. Not used by the other targets                                                                                                                                              |
| `--publish_schema_pattern PATTERN` | Target schema name, with tokens `{schema}`, `{database}`, `{date}`, `{timestamp}`, `{uuid}`, `{subpath}` (`{table}` is not allowed). `{database}` is the Snowflake target database, and empty for the other targets. Default: `EXT_{schema}` (external), `{schema}` (internal) |
| `--publish_table_pattern PATTERN`  | Target table name, same tokens plus `{table}`, which it must include. Default: `{table}`; Fabric Lakehouse: `{schema}_{table}` (internal) or `VW_{schema}_{table}` (external), with only `{schema}` and `{table}` substituted                                                  |
| `--no_views`                       | Do not create views over external tables (Snowflake)                                                                                                                                                                                                                           |
| `--pk_constraints`                 | Add the source primary keys to internal tables (Snowflake, Databricks, Fabric Warehouse, MotherDuck, Redshift; not applied on BigQuery)                                                                                                                                        |

Platform-specific defaults and behavior are described on each [target platform](/latest/integrations/snowflake-publishing.md) page.

### Metadata[​](#metadata "Direct link to Metadata")

| Option                 | Description                                                                           |
| ---------------------- | ------------------------------------------------------------------------------------- |
| `--generate_metadata`  | Write CDM metadata (`manifest.json` and `.cdm.json` files) next to the Parquet files  |
| `--manifest_name NAME` | Name of the CDM manifest. Default: schema name (per schema) or database name (global) |

### Error Handling[​](#error-handling "Direct link to Error Handling")

| Option                  | Description                                                                                                 |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| `--error_action ACTION` | `fail` (default): stop at the first failed table. `continue`: log the error and go on with the other tables |

## config list[​](#config-list "Direct link to config list")

List the sync configurations.

```
usage: LakeXpress config list --auth_file AUTH_FILE
                              --lxdb_auth_id ID [--env_name NAME]
                              [common options]
```

| Option            | Description                                      |
| ----------------- | ------------------------------------------------ |
| `--env_name NAME` | Only list the configurations of this environment |

## config delete[​](#config-delete "Direct link to config delete")

Delete a sync configuration with its run history, table metadata and watermarks. Exported files and published tables are **not** deleted.

```
usage: LakeXpress config delete --auth_file AUTH_FILE
                                --lxdb_auth_id ID --sync_id SYNC_ID
                                [--confirm] [common options]
```

| Option         | Description                                        |
| -------------- | -------------------------------------------------- |
| `--sync_id ID` | **Required.** Sync configuration to delete         |
| `--confirm`    | Delete it. Without it, shows what would be deleted |


---

# lxdb

Manage the LakeXpress DB, the database where LakeXpress stores its configurations and run history. How to create it: [Set up the LakeXpress DB](/latest/lakexpress-db/.md). Examples: [LakeXpress DB recipes](/latest/guides/recipes/lakexpress-db.md).

All `lxdb` commands also accept the [common options](/latest/cli/.md#common-options).

Destructive commands are dry runs by default

`drop`, `truncate` and `release-locks` only show what they would do. Nothing changes until you add `--confirm`.

## lxdb init[​](#lxdb-init "Direct link to lxdb init")

Create the LakeXpress tables in the LakeXpress DB, and check the connection. Existing tables and data are kept, so it is safe to run again. The first `config create` or `sync` also creates the tables if needed.

```
usage: LakeXpress lxdb init --auth_file AUTH_FILE --lxdb_auth_id ID
                            [common options]
```

No options besides the common ones.

## lxdb drop[​](#lxdb-drop "Direct link to lxdb drop")

Drop all LakeXpress tables from the LakeXpress DB, with their data.

```
usage: LakeXpress lxdb drop --auth_file AUTH_FILE --lxdb_auth_id ID
                            [--confirm] [common options]
```

| Option      | Description                                                         |
| ----------- | ------------------------------------------------------------------- |
| `--confirm` | Drop the tables. Without it, lists the tables that would be dropped |

## lxdb truncate[​](#lxdb-truncate "Direct link to lxdb truncate")

Delete all data from the LakeXpress tables (configurations, runs, watermarks) and keep the tables.

```
usage: LakeXpress lxdb truncate --auth_file AUTH_FILE
                                --lxdb_auth_id ID [--confirm]
                                [--sync_id SYNC_ID] [common options]
```

| Option         | Description                                                                                                                                                          |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--confirm`    | Delete the data. Without it, lists the tables that would be emptied                                                                                                  |
| `--sync_id ID` | Not implemented yet: with this option, nothing is deleted. To remove one sync configuration and its data, use [`config delete`](/latest/cli/config.md#config-delete) |

## lxdb locks[​](#lxdb-locks "Direct link to lxdb locks")

List the tables currently locked in the LakeXpress DB.

Locks are only used by **incremental syncs**: they prevent two runs from changing the same watermark at the same time. Full syncs do not use locks.

```
usage: LakeXpress lxdb locks --auth_file AUTH_FILE --lxdb_auth_id ID
                             [--sync_id SYNC_ID] [common options]
```

| Option         | Description                                    |
| -------------- | ---------------------------------------------- |
| `--sync_id ID` | Only show the locks of this sync configuration |

## lxdb release-locks[​](#lxdb-release-locks "Direct link to lxdb release-locks")

Release locks left behind by a sync that crashed or was killed. They block the next runs of the same tables. Use `lxdb locks` first to see them.

```
usage: LakeXpress lxdb release-locks --auth_file AUTH_FILE
                                     --lxdb_auth_id ID
                                     [--max_age_hours HOURS]
                                     [--table_id ID] [--confirm]
                                     [common options]
```

| Option              | Description                                                           |
| ------------------- | --------------------------------------------------------------------- |
| `--max_age_hours N` | Release the locks older than N hours (default: 24)                    |
| `--table_id ID`     | Release the lock of this table only (IDs are shown by `lxdb locks`)   |
| `--confirm`         | Release the locks. Without it, shows the locks that would be released |


---

# run

Run an export from a YAML configuration file. This is the legacy way of running LakeXpress. Each `sync` run still writes a copy of its configuration as a YAML file next to its log file (`lx_<date>_<time>_<run_id>.yml`), but configurations stored in the LakeXpress DB and run with [`sync`](/latest/cli/sync.md) are the recommended way.

```
usage: LakeXpress run --config CONFIG [--auth_file AUTH_FILE]
                      [--lxdb_auth_id ID] [common options]
```

| Option                 | Description                                              |
| ---------------------- | -------------------------------------------------------- |
| `-c, --config PATH`    | **Required.** YAML configuration file                    |
| `-a, --auth_file PATH` | Credentials file, overriding the one in the YAML file    |
| `--lxdb_auth_id ID`    | LakeXpress DB entry, overriding the one in the YAML file |

Also accepts the other [common options](/latest/cli/.md#common-options).


---

# status

Show the runs of the sync configurations and their status (completed, failed, running), read from the LakeXpress DB.

```
usage: LakeXpress status --auth_file AUTH_FILE --lxdb_auth_id ID
                         (--sync_id SYNC_ID | --run_id RUN_ID)
                         [--verbose] [common options]
```

| Option          | Description                                      |
| --------------- | ------------------------------------------------ |
| `--sync_id ID`  | Show the runs of this sync configuration         |
| `--run_id ID`   | Show the details of this run                     |
| `-v, --verbose` | List up to the 20 most recent runs (default: 10) |

One of `--sync_id` or `--run_id` is required: without them, `status` exits with code 10. To list the sync configurations, use [`config list`](/latest/cli/config.md).

Also accepts the [common options](/latest/cli/.md#common-options).


---

# sync

Run a sync configuration created by [`config create`](/latest/cli/config.md#config-create): export the tables, then publish them if the configuration has a publish target. Examples: [Examples & Recipes](/latest/guides/command-cookbook.md), [Resume Failed Exports](/latest/guides/recipes/resume-failed-exports.md).

All `sync` commands also accept the [common options](/latest/cli/.md#common-options).

## sync[​](#sync-command "Direct link to sync")

Export, then publish.

```
usage: LakeXpress sync --auth_file AUTH_FILE --lxdb_auth_id ID
                       --sync_id SYNC_ID [--resume]
                       [--run_id RUN_ID] [--fastbcp_dir_path PATH]
                       [common options]
```

| Option                    | Description                                                                                   |
| ------------------------- | --------------------------------------------------------------------------------------------- |
| `--sync_id ID`            | **Required.** Sync configuration to run (printed by `config create`, listed by `config list`) |
| `--resume`                | Resume an incomplete run instead of starting a new one: tables already exported are skipped   |
| `--run_id ID`             | Run to resume, with `--resume` (run IDs are shown in the logs and by `status`)                |
| `--fastbcp_dir_path PATH` | Use this FastBCP folder for this run instead of the one saved in the configuration            |

## sync\[export][​](#sync-export "Direct link to sync\[export]")

Export only, without publishing. Useful to publish later with `sync[publish]`.

```
usage: LakeXpress sync[export] --auth_file AUTH_FILE
                               --lxdb_auth_id ID --sync_id SYNC_ID
                               [--fastbcp_dir_path PATH]
                               [common options]
```

| Option                    | Description                                                                        |
| ------------------------- | ---------------------------------------------------------------------------------- |
| `--sync_id ID`            | **Required.** Sync configuration to export                                         |
| `--fastbcp_dir_path PATH` | Use this FastBCP folder for this run instead of the one saved in the configuration |

In most shells, quote the command name: `'sync[export]'`.

## sync\[publish][​](#sync-publish "Direct link to sync\[publish]")

Publish files already exported by a previous run, without exporting again.

```
usage: LakeXpress sync[publish] --auth_file AUTH_FILE
                                --lxdb_auth_id ID --sync_id SYNC_ID
                                [--run_id RUN_ID] [common options]
```

| Option         | Description                                                    |
| -------------- | -------------------------------------------------------------- |
| `--sync_id ID` | **Required.** Sync configuration to publish                    |
| `--run_id ID`  | Run whose files to publish (default: the latest completed run) |

In most shells, quote the command name: `'sync[publish]'`.


---

# Database Connections

LakeXpress uses two distinct database roles, each requiring its own connection entry in the credentials file:

* **LakeXpress DB** — An internal database used by LakeXpress to store configuration, track sync state, and log pipeline runs. This is not a source of data; it is LakeXpress's own bookkeeping store. You point LakeXpress at it with `--lxdb_auth_id`. It is set up once, at installation: see [Set up the LakeXpress DB](/latest/lakexpress-db/.md).
* **Source Database** — The database you want to extract data from (e.g., a production PostgreSQL or Oracle instance). You point LakeXpress at it with `--source_db_auth_id`.

Both connections are defined in the same credentials JSON file, distinguished by the key name you give each entry.

## Supported Databases[​](#supported-databases "Direct link to Supported Databases")

### Source Databases[​](#source-databases "Direct link to Source Databases")

LakeXpress reads tables from these databases. Each page covers the connection settings and the privileges the export user needs.

* [![](/img/logos/postgresql.svg)PostgreSQL](/latest/configuration/databases/postgresql.md)Ctid-based parallel export
* [![](/img/logos/oracle.svg)Oracle](/latest/configuration/databases/oracle.md)thin and thick modes, Rowid-based parallel export
* [![](/img/logos/sqlserver.svg)SQL Server](/latest/configuration/databases/sql-server.md)Physloc-based parallel export
* [![](/img/logos/mysql.svg)MySQL](/latest/configuration/databases/mysql.md)partition detection
* [![](/img/logos/mariadb.svg)MariaDB](/latest/configuration/databases/mariadb.md)dedicated `mariadb` ds\_type
* [![](/img/logos/teradata.svg)Teradata](/latest/configuration/databases/teradata.md)requires the `teradatasql` driver
* [![](/img/logos/sap.svg)SAP HANA](/latest/configuration/databases/sap-hana.md)requires `hdbcli`, and HDBDOTNETCORE for FastBCP

### ADBC Extraction[​](#adbc-extraction "Direct link to ADBC Extraction")

FastBCP extracts SQL Server, PostgreSQL and Teradata through the arpe.io ADBC drivers it embeds, by default. Set `"use_adbc": false` on a source, at the same level as `ds_type`, to use FastBCP's native driver instead. Oracle has an ADBC driver too, used only with `"use_adbc": true` (see [Oracle](/latest/configuration/databases/oracle.md#adbc-export)).

| Source         | Default extraction     | Other choice                                  |
| -------------- | ---------------------- | --------------------------------------------- |
| SQL Server     | `adbc_mssql` (ADBC)    | `mssql` (native), with `"use_adbc": false`    |
| PostgreSQL     | `adbc_pgsql` (ADBC)    | `pgcopy` (native), with `"use_adbc": false`   |
| Oracle         | `oraodp` (native)      | `adbc_oracle` (ADBC), with `"use_adbc": true` |
| Teradata       | `adbc_teradata` (ADBC) | `teradata` (native), with `"use_adbc": false` |
| MySQL, MariaDB | `mysql` (native)       | none                                          |
| SAP HANA       | `hana` (native)        | none                                          |

* ADBC only changes the extraction to Parquet. LakeXpress still discovers schemas, tables and columns with its own Python drivers.
* MySQL, MariaDB and SAP HANA have no ADBC driver. `"use_adbc": true` on them logs a warning and uses the native driver.
* `use_adbc` must be `true` or `false`; any other value stops the sync before the export with exit code 10.
* The ADBC drivers need the ADBC block of the license file (exit code 13 without it), and ship for linux-x64 and win-x64.
* ADBC writes timezone-aware timestamps as UTC and keeps microseconds. See the ADBC section of each source page for the type differences with the native drivers.

### LakeXpress DB[​](#lakexpress-db "Direct link to LakeXpress DB")

The database where LakeXpress keeps its configurations and run history. How to create it and choose a backend: [Set up the LakeXpress DB](/latest/lakexpress-db/.md).

* [![](/img/logos/postgresql.svg)PostgreSQL](/latest/lakexpress-db/postgresql.md)
* [![](/img/logos/sqlserver.svg)SQL Server](/latest/lakexpress-db/sql-server.md)
* [![](/img/logos/mysql.svg)MySQL](/latest/lakexpress-db/mysql.md)
* [SQLite](/latest/lakexpress-db/sqlite.md)
* [![](/img/logos/duckdb.svg)DuckDB](/latest/lakexpress-db/duckdb.md)

## Credentials File Format[​](#credentials-file-format "Direct link to Credentials File Format")

All database credentials are stored in a single JSON file, `credentials.json` in this documentation (any name works: pass it with `-a`). Each top-level key is a connection identifier you choose — you reference these identifiers on the command line via `--lxdb_auth_id` and `--source_db_auth_id`. The file can hold as many connection entries as you need.

Create a JSON file with your database credentials:

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_oracle": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-server.com",
      "port": 1521,
      "database": "orclpdb1",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  },
  "lxdb_sqlite": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/path/to/lakexpress_log.sqlite"
    }
  }
}
```

Every command takes the file with `-a` and picks entries by name: `--lxdb_auth_id` for the LakeXpress DB, `--source_db_auth_id` for the source database. For complete commands, see the [Getting Started](/latest/guides/quickstart.md) guide and [Examples & Recipes](/latest/guides/command-cookbook.md).

## Publishing Credentials[​](#publishing-credentials "Direct link to Publishing Credentials")

Publishing targets use their own credential entries in the same `credentials.json` file. See each publishing guide for authentication setup:

* [![](/img/logos/snowflake.svg)Snowflake](/latest/integrations/snowflake-publishing.md)
* [![](/img/logos/databricks.svg)Databricks](/latest/integrations/databricks-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric](/latest/integrations/fabric-publishing.md)
* [![](/img/logos/bigquery.svg)BigQuery](/latest/integrations/bigquery-publishing.md)
* [![](/img/logos/motherduck.svg)MotherDuck](/latest/integrations/motherduck-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue](/latest/integrations/glue-publishing.md)
* [Amazon Redshift](/latest/integrations/redshift-publishing.md)
* [![](/img/logos/ducklake.svg)DuckLake](/latest/integrations/ducklake-publishing.md)

## Security Best Practices[​](#security-best-practices "Direct link to Security Best Practices")

Credential storage

* Never commit credentials to version control
* Restrict `credentials.json` permissions:
  <!-- -->
  ```
  chmod 600 credentials.json
  ```
* Use a separate credentials file per environment (dev, staging, prod)

Least privilege

* Create dedicated database users for LakeXpress
* Grant only `SELECT` on source databases: LakeXpress never writes to them
* Grant full privileges on the LakeXpress DB: LakeXpress creates and updates its own tracking tables there

Network security

* Use SSL/TLS connections
* Restrict database access by IP
* Use VPN or private networks

Password management

* Use strong passwords (12+ characters, mixed case, numbers, symbols)
* Rotate passwords regularly
* Consider secret managers (AWS Secrets Manager, HashiCorp Vault), or the environment variables below

### Environment Variables for Credentials[​](#environment-variables-for-credentials "Direct link to Environment Variables for Credentials")

LakeXpress supports `$env{VAR_NAME}` syntax in any string value in the credentials JSON. An error is raised if a referenced variable is not set.

```
{
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  }
}
```

Set variables before running LakeXpress:

* Windows (PowerShell)
* Windows (cmd)
* Linux

```
$env:LX_PG_USER = "dataexport"
$env:LX_PG_PASSWORD = "SecureP@ssw0rd"
$env:LX_MSSQL_USER = "sa"
$env:LX_MSSQL_PASSWORD = "StrongP@ssw0rd"
```

```
set LX_PG_USER=dataexport
set LX_PG_PASSWORD=SecureP@ssw0rd
set LX_MSSQL_USER=sa
set LX_MSSQL_PASSWORD=StrongP@ssw0rd
```

```
export LX_PG_USER="dataexport"
export LX_PG_PASSWORD="SecureP@ssw0rd"
export LX_MSSQL_USER="sa"
export LX_MSSQL_PASSWORD="StrongP@ssw0rd"
```

Plain-text values still work -- values without `$env{` are returned as-is.

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/latest/guides/quickstart.md)Getting started
* [CLI Reference](/latest/cli/.md)Command-line options
* [Intermediate Storage](/latest/configuration/storage/storage-overview.md)Storage configuration


---

# MariaDB

MariaDB is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Although MariaDB is derived from MySQL, use `ds_type: "mariadb"` rather than `"mysql"` so LakeXpress can label and track the source correctly. Under the hood, `mariadb` connections are routed through the same MySQL connector (`mysql-connector-python`) as `ds_type: "mysql"` — there is no separate MariaDB driver.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                                                                                                                                                                |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"mariadb"`                                                                                                                                                        |
| `auth_mode` | Yes      | Must be `"classic"`                                                                                                                                                        |
| `server`    | No       | Hostname or IP (`host` is also accepted; default: `localhost`)                                                                                                             |
| `port`      | No       | Port (default: 3306)                                                                                                                                                       |
| `database`  | No       | Default database/schema to connect to. If omitted, LakeXpress connects without selecting a default database (useful when discovery is scoped by job configuration instead) |
| `username`  | Yes      | MariaDB username                                                                                                                                                           |
| `password`  | Yes      | MariaDB password                                                                                                                                                           |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON your_database.* TO 'lakexpress_user'@'%';

-- INFORMATION_SCHEMA access is granted by default
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_mariadb": {
    "ds_type": "mariadb",
    "auth_mode": "classic",
    "info": {
      "server": "localhost",
      "port": 3306,
      "database": "your-database",
      "username": "$env{LX_MARIADB_USER}",
      "password": "$env{LX_MARIADB_PASSWORD}"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Access denied for user:** Verify credentials, check host access: `SELECT user, host FROM mysql.user WHERE user='your_user';`, grant if needed: `GRANT ALL PRIVILEGES ON database.* TO 'user'@'%';`

**Unknown database:** Verify name: `SHOW DATABASES;`, create if needed: `CREATE DATABASE your_database;`

**Connection refused:** Check MariaDB is running (`systemctl status mariadb`), verify port (`netstat -tlnp | grep 3306`), check `bind-address` allows remote connections, check firewall (`sudo ufw allow 3306/tcp`).

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# MySQL

This page covers MySQL as a **source database**, the database LakeXpress exports data from. To use MySQL as the LakeXpress DB, see [MySQL as LakeXpress DB](/latest/lakexpress-db/mysql.md).

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                                                                                                                                                                |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"mysql"`                                                                                                                                                          |
| `auth_mode` | Yes      | Must be `"classic"`                                                                                                                                                        |
| `server`    | No       | Hostname or IP (`host` is also accepted; default: `localhost`)                                                                                                             |
| `port`      | No       | Port (default: 3306)                                                                                                                                                       |
| `database`  | No       | Default database/schema to connect to. If omitted, LakeXpress connects without selecting a default database (useful when discovery is scoped by job configuration instead) |
| `username`  | Yes      | MySQL username                                                                                                                                                             |
| `password`  | Yes      | MySQL password                                                                                                                                                             |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

LakeXpress only needs read access to the databases it exports:

```
GRANT SELECT ON your_database.* TO 'lakexpress_user'@'%';
```

`INFORMATION_SCHEMA` access is granted by default.

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_mysql": {
    "ds_type": "mysql",
    "auth_mode": "classic",
    "info": {
      "server": "localhost",
      "port": 3306,
      "database": "your-database",
      "username": "$env{LX_MYSQL_USER}",
      "password": "$env{LX_MYSQL_PASSWORD}"
    }
  }
}
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Access denied for user:** Verify credentials, check host access: `SELECT user, host FROM mysql.user WHERE user='your_user';`, grant if needed: `GRANT ALL PRIVILEGES ON database.* TO 'user'@'%';`

**Unknown database:** Verify name: `SHOW DATABASES;`, create if needed: `CREATE DATABASE your_database;`

**Connection refused:** Check MySQL is running (`systemctl status mysql`), verify port (`netstat -tlnp | grep 3306`), check `bind-address` allows remote connections, check firewall (`sudo ufw allow 3306/tcp`).

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# Oracle

Oracle is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required | Description                                                                                                                                                              |
| ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ds_type`   | Yes      | Must be `"oracle"`                                                                                                                                                       |
| `auth_mode` | Yes      | Must be `"classic"`                                                                                                                                                      |
| `username`  | Yes      | Oracle username                                                                                                                                                          |
| `password`  | Yes      | Oracle password                                                                                                                                                          |
| `server`    | No       | Hostname or IP (`host` is also accepted; default: `localhost`)                                                                                                           |
| `port`      | No       | Port (default: 1521)                                                                                                                                                     |
| `database`  | Yes      | Service name (a SID is not accepted: LakeXpress connects with `service_name`)                                                                                            |
| `lib_dir`   | No       | Client library directory (thick mode)                                                                                                                                    |
| `use_adbc`  | No       | Export through FastBCP's ADBC driver (default: `false`, FastBCP's native driver). Set at the same level as `ds_type`, not inside `info`. See [ADBC Export](#adbc-export) |

## Thin vs Thick Mode[​](#thin-vs-thick-mode "Direct link to Thin vs Thick Mode")

**Thin Mode (Recommended):**

* No client libraries required
* Oracle 12.1+
* Omit `lib_dir`

**Thick Mode:**

* Required for Oracle 11.2 and earlier
* Requires Oracle Instant Client
* Set `lib_dir` to the Instant Client directory

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

LakeXpress uses RowID-based parallel exports to read Oracle tables efficiently. The database user needs catalog access for metadata discovery (table structure, RowID ranges, and export method selection):

```
-- As Oracle DBA (e.g., sys as sysdba)
GRANT SELECT_CATALOG_ROLE TO your_username;
```

## Example: Thin Mode[​](#example-thin-mode "Direct link to Example: Thin Mode")

```
{
  "oracle_prod": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-prod.company.com",
      "port": 1521,
      "database": "PRODPDB"
    }
  }
}
```

## Example: Thick Mode[​](#example-thick-mode "Direct link to Example: Thick Mode")

```
{
  "oracle_legacy": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-11g.company.com",
      "port": 1521,
      "database": "LEGACY",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  }
}
```

## ADBC Export[​](#adbc-export "Direct link to ADBC Export")

With `"use_adbc": true`, FastBCP reads the source through the arpe.io ADBC driver for Oracle (connection type `adbc_oracle`) instead of its native ODP.NET driver. It needs no Oracle Instant Client, whatever `lib_dir` says: `lib_dir` only applies to LakeXpress's own connection. ADBC is not the default for Oracle yet: the driver writes `NUMBER` columns without precision as text, while the publish targets declare them `decimal(38,18)`. Use it for sources without such columns, or exports you do not publish. The setting only changes the export: LakeXpress still discovers schemas and tables with its own driver. The run log shows the path once per run, e.g. `Source extraction: adbc_oracle (ADBC)`.

```
{
  "oracle_adbc": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "use_adbc": true,
    "info": {
      "username": "$env{LX_ORA_USER}",
      "password": "$env{LX_ORA_PASSWORD}",
      "server": "oracle-prod.company.com",
      "port": 1521,
      "database": "ORCLPDB1"
    }
  }
}
```

Network encryption and data integrity default to `accepted`. To change them, set the `FASTBCP_ADBC_ENCRYPTION` and `FASTBCP_ADBC_DATA_INTEGRITY` environment variables before running LakeXpress.

The Parquet files written through ADBC differ from the native ones:

* `NUMBER` without precision is text (the native driver writes `decimal(38,18)`).
* `NUMBER(p)` with `p` up to 9 is a 64-bit integer (the native driver writes a 32-bit integer).
* `TIMESTAMP` keeps microseconds (the native driver writes milliseconds).
* `TIMESTAMP WITH TIME ZONE` is a UTC timestamp (the native driver writes text such as `1/1/1970 12:00:00 AM +00:00`).
* `DATE` and `TIMESTAMP` before 1582-10-15, which Oracle stores on the Julian calendar, are converted to the same day on the Gregorian calendar: `0001-01-01` becomes `0000-12-30`. The native driver keeps the label, which is a different day.

The ADBC driver needs its own license. It is included in the license file that arpe.io sends for LakeXpress, and LakeXpress passes it to FastBCP. If your license file has no ADBC block, provide the driver license with `ARPEIO_ADBC_LICENCE` (license text) or `ARPEIO_ADBC_LICENCE_FILE` (path). Without either, the sync stops before the export with exit code 13 (`LICENSE_ERROR`). A license file issued before 2026-10-26 has no FastBCP block: FastBCP then uses its own `FastBCP.lic`, which must carry the ADBC block. See [License Information](/latest/installation.md#license-information).

ADBC drivers ship with FastBCP for linux-x64 and win-x64. See [ADBC Extraction](/latest/configuration/database-configuration.md#adbc-extraction) for the other sources.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**ORA-12154: TNS could not resolve connect identifier:** Verify service name, check `tnsnames.ora`, try IP instead of hostname, confirm listener is running.

**ORA-28000: account is locked:**

```
ALTER USER your_username ACCOUNT UNLOCK;
```

**Thick mode library not found:** Verify `lib_dir` path, confirm Instant Client is installed, check file permissions, set `LD_LIBRARY_PATH` on Linux.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# PostgreSQL

This page covers PostgreSQL as a **source database**, the database LakeXpress exports data from. To use PostgreSQL as the LakeXpress DB, see [PostgreSQL as LakeXpress DB](/latest/lakexpress-db/postgresql.md).

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter                   | Required     | Description                                                                                                                                                                                                                       |
| --------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ds_type`                   | Yes          | Must be `"postgres"`                                                                                                                                                                                                              |
| `auth_mode`                 | Yes          | `"classic"` (username/password) or `"connection_string"` (supply a full connection string)                                                                                                                                        |
| `username`                  | Classic only | PostgreSQL username                                                                                                                                                                                                               |
| `password`                  | Classic only | PostgreSQL password                                                                                                                                                                                                               |
| `server`                    | Classic only | Hostname or IP (`host` is also accepted)                                                                                                                                                                                          |
| `port`                      | No           | Port (default: 5432)                                                                                                                                                                                                              |
| `database`                  | Classic only | Database name                                                                                                                                                                                                                     |
| `sslmode`                   | No           | PostgreSQL SSL mode, e.g. `disable`, `prefer`, `require`, `verify-full` (default: `prefer`)                                                                                                                                       |
| `use_adbc`                  | No           | Export through FastBCP's ADBC driver (default: `true`). `false` uses FastBCP's native driver. Set at the same level as `ds_type`, not inside `info`. See [ADBC Export](#adbc-export)                                              |
| `fastbcp_connection_string` | No           | Connection string passed to FastBCP as is, in the format of its Npgsql (.NET) driver. Overrides the one LakeXpress builds; use it for options that do not convert (see [Connection String Mode](#example-connection-string-mode)) |

With `auth_mode: "connection_string"`, only the `connection_string` field is used; `username`, `password`, `server`, `port`, `database`, and `sslmode` are ignored.

The export uses the same `sslmode` as LakeXpress's own connection.

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

LakeXpress only needs read access to the schemas it exports:

```
GRANT USAGE ON SCHEMA public TO lakexpress_user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO lakexpress_user;
```

`information_schema` access is granted by default.

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "pg-prod.company.com",
      "port": 5432,
      "database": "analytics"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

```
{
  "postgres_cs": {
    "ds_type": "postgres",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "host=pg-prod.company.com port=5432 dbname=analytics user=$env{LX_PG_USER} password=$env{LX_PG_PASSWORD} sslmode=require"
    }
  }
}
```

The string can also be a URI (`postgresql://user:password@host:5432/dbname?sslmode=require`). LakeXpress converts it for FastBCP's Npgsql driver: `host`, `port`, `dbname`, `user`, `password`, `sslmode`, `sslrootcert`, `application_name` and `connect_timeout` are passed. Other keywords are not passed to FastBCP, and LakeXpress logs a warning; set `fastbcp_connection_string` to pass them in Npgsql form.

## ADBC Export[​](#adbc-export "Direct link to ADBC Export")

By default, FastBCP reads the source through the arpe.io ADBC driver (ArrowFEBE, connection type `adbc_pgsql`). Set `"use_adbc": false` to use FastBCP's native driver instead, for example to work around a driver issue. The setting only changes the export: LakeXpress still discovers schemas and tables with its own driver, and all `auth_mode` values work as before. The run log shows the path once per run, e.g. `Source extraction: adbc_pgsql (ADBC)`.

```
{
  "postgres_native": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "use_adbc": false,
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "pg-prod.company.com",
      "port": 5432,
      "database": "analytics"
    }
  }
}
```

The Parquet files written through ADBC differ from the native ones:

* `time` and `timestamp` keep microseconds (the native driver writes milliseconds).
* `timestamptz` is a UTC timestamp (the native driver writes a timestamp without time zone).
* `jsonb` is text (the native driver writes binary).

The ADBC driver needs its own license. It is included in the license file that arpe.io sends for LakeXpress, and LakeXpress passes it to FastBCP. If your license file has no ADBC block, provide the driver license with `ARPEIO_ADBC_LICENCE` (license text), `ARPEIO_ADBC_LICENCE_FILE` (path), or `arpeio.adbc.license_file=<path>` in `info.fastbcp_connection_string` (the connection string LakeXpress builds from `connection_string` does not carry it). Without any of these, the sync stops before the export with exit code 13 (`LICENSE_ERROR`). See [License Information](/latest/installation.md#license-information).

ADBC drivers ship with FastBCP for linux-x64 and win-x64. See [ADBC Extraction](/latest/configuration/database-configuration.md#adbc-extraction) for the other sources.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Connection refused:** Check PostgreSQL is running (`systemctl status postgresql`), verify `pg_hba.conf` allows your IP, check firewall (`sudo ufw allow 5432/tcp`).

**Authentication failed:** Verify credentials, check password encryption in `pg_hba.conf`, confirm user exists: `SELECT * FROM pg_user WHERE usename='your_user';`

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# SAP HANA

SAP HANA is supported as a **source database** — the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Connecting to SAP HANA requires the `hdbcli` Python driver. If you are using FastBCP for high-throughput exports, the HDBDOTNETCORE runtime is also required (see the note at the bottom of this page).

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required     | Description                                                                                             |
| ----------- | ------------ | ------------------------------------------------------------------------------------------------------- |
| `ds_type`   | Yes          | Must be `"saphana"`                                                                                     |
| `auth_mode` | Yes          | `"classic"` for individual fields; `"connection_string"` to supply a full ODBC connection string        |
| `server`    | No           | Hostname or IP (`host` is also accepted; default: `localhost`)                                          |
| `port`      | No           | Port (default: 39017)                                                                                   |
| `database`  | No           | Tenant database name. Only relevant for multi-tenant (MDC) HANA systems; omit for single-tenant systems |
| `username`  | Classic only | SAP HANA username                                                                                       |
| `password`  | Classic only | SAP HANA password                                                                                       |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON SCHEMA your_schema TO lakexpress_user;

-- Metadata access:
GRANT SELECT ON SYS.TABLES TO lakexpress_user;
GRANT SELECT ON SYS.COLUMNS TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_saphana": {
    "ds_type": "saphana",
    "auth_mode": "classic",
    "info": {
      "server": "hana-prod.company.com",
      "port": 39017,
      "database": "HDB",
      "username": "$env{LX_HANA_USER}",
      "password": "$env{LX_HANA_PASSWORD}"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

Use `auth_mode: "connection_string"` when you need to pass advanced ODBC driver options, or when your environment already manages connection strings centrally. The entire connection is expressed as a single `connection_string` value; the individual `server`, `port`, `database`, `username`, and `password` fields are not used.

```
{
  "source_saphana_cs": {
    "ds_type": "saphana",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "DRIVER={HDBODBC};SERVERNODE=$env{LX_HANA_HOST}:39017;DATABASE=$env{LX_HANA_DB};UID=$env{LX_HANA_USER};PWD=$env{LX_HANA_PASSWORD}"
    }
  }
}
```

FastBCP uses this string as is. For its own connection, LakeXpress turns it into `hdbcli` arguments: `SERVERNODE` gives the host and port (a failover list such as `hana1:30015,hana2:30015` is accepted), `UID` the user, `PWD` the password and `DATABASE` the tenant database; `DRIVER` is ignored and other keywords are passed as `hdbcli` connection properties.

note

FastBCP requires the HDBDOTNETCORE runtime. Set the `HDBDOTNETCORE` environment variable to the runtime directory path before running LakeXpress.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# SQL Server

This page covers SQL Server as a **source database**, the database LakeXpress exports data from. To use SQL Server as the LakeXpress DB, see [SQL Server as LakeXpress DB](/latest/lakexpress-db/sql-server.md).

note

LakeXpress uses the ODBC driver for SQL Server connections. If ODBC drivers are not installed on the host machine, LakeXpress automatically falls back to PyMSSQL.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter                   | Required           | Description                                                                                                                                                                                                                          |
| --------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ds_type`                   | Yes                | Must be `"mssql"`                                                                                                                                                                                                                    |
| `auth_mode`                 | Yes                | `"classic"` (username/password), `"integrated"` (Windows authentication), or `"connection_string"` (supply a full ODBC connection string)                                                                                            |
| `username`                  | Classic only       | SQL Server username                                                                                                                                                                                                                  |
| `password`                  | Classic only       | SQL Server password                                                                                                                                                                                                                  |
| `server`                    | Classic/integrated | Hostname or IP (`host` is also accepted)                                                                                                                                                                                             |
| `port`                      | No                 | Port (default: 1433)                                                                                                                                                                                                                 |
| `database`                  | Classic/integrated | Database name                                                                                                                                                                                                                        |
| `encrypt`                   | No                 | Enable connection encryption (default: `true`)                                                                                                                                                                                       |
| `trust_server_certificate`  | No                 | Trust the server's TLS certificate without validation (default: `false`). Set to `true` for self-signed certificates, common on test servers and `localhost`                                                                         |
| `application_intent`        | No                 | `"ReadWrite"` (default) or `"ReadOnly"`. `"ReadOnly"` lets an Always On listener with read-only routing send both LakeXpress and the export to a readable secondary. See [Availability Groups](#availability-groups)                 |
| `use_adbc`                  | No                 | Export through FastBCP's ADBC driver (default: `true`). `false` uses FastBCP's native driver. Set at the same level as `ds_type`, not inside `info`. See [ADBC Export](#adbc-export)                                                 |
| `fastbcp_connection_string` | No                 | Connection string passed to FastBCP as is, in the format of its SqlClient (.NET) driver. Overrides the one LakeXpress builds; use it for options that do not convert (see [Connection String Mode](#example-connection-string-mode)) |

With `auth_mode: "connection_string"`, only the `connection_string` field is used; `username`, `password`, `server`, `port`, `database`, `encrypt`, `trust_server_certificate` and `application_intent` are ignored. Set `ApplicationIntent` in the string instead.

The export uses the same TLS settings as LakeXpress's own connection: `encrypt` and `trust_server_certificate` (or `Encrypt` and `TrustServerCertificate` in the connection string) also apply to FastBCP. With the defaults, the export verifies the server certificate too.

### Availability Groups[​](#availability-groups "Direct link to Availability Groups")

LakeXpress's own connection (discovery and the incremental `MAX(column)` query) and the export use the same `application_intent`, so they read the same replica. With the default `"ReadWrite"`, both read the primary.

With `"ReadOnly"`, both are routed to a readable secondary. The incremental upper bound then comes from the secondary, so it never gets ahead of the rows the export can see. If the listener routes connections across several secondaries, LakeXpress and the export may land on different ones; add an `--incremental_safety_lag` larger than the replica lag, or point `server` at a single secondary.

If you set `fastbcp_connection_string`, give it the same `ApplicationIntent` as LakeXpress's connection.

With `auth_mode: "integrated"`, Windows integrated authentication is used and `username`/`password` are omitted.

note

The `pymssql` fallback driver (used automatically when no ODBC driver is installed) only supports `auth_mode: "classic"` and `application_intent: "ReadWrite"`. `"integrated"` and `"connection_string"` require the ODBC driver.

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

LakeXpress only needs read access to the schemas it exports:

```
GRANT SELECT ON SCHEMA::dbo TO lakexpress_user;
GRANT VIEW DEFINITION ON SCHEMA::dbo TO lakexpress_user;
```

`INFORMATION_SCHEMA` access is granted by default.

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-prod.company.com",
      "port": 1433,
      "database": "SalesDB"
    }
  }
}
```

## Example: Integrated Authentication[​](#example-integrated-authentication "Direct link to Example: Integrated Authentication")

```
{
  "mssql_integrated": {
    "ds_type": "mssql",
    "auth_mode": "integrated",
    "info": {
      "server": "mssql-prod.company.com",
      "port": 1433,
      "database": "SalesDB"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

```
{
  "mssql_cs": {
    "ds_type": "mssql",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "DRIVER={ODBC Driver 18 for SQL Server};SERVER=mssql-prod.company.com,1433;DATABASE=SalesDB;UID=$env{LX_MSSQL_USER};PWD=$env{LX_MSSQL_PASSWORD};Encrypt=yes;TrustServerCertificate=yes;"
    }
  }
}
```

LakeXpress uses this ODBC string for its own connection and passes it to FastBCP without `DRIVER`, which FastBCP's SqlClient driver does not accept. A `DSN=` string cannot be passed to FastBCP: set `fastbcp_connection_string` to a SqlClient connection string (`Server=...;Database=...;User Id=...;Password=...`).

## ADBC Export[​](#adbc-export "Direct link to ADBC Export")

By default, FastBCP reads the source through the arpe.io ADBC driver (ArrowTDS, connection type `adbc_mssql`). Set `"use_adbc": false` to use FastBCP's native driver instead, for example to work around a driver issue. The setting only changes the export: LakeXpress still discovers schemas and tables with its own driver, and all `auth_mode` values work as before. The run log shows the path once per run, e.g. `Source extraction: adbc_mssql (ADBC)`.

```
{
  "sql_server_native": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "use_adbc": false,
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-prod.company.com",
      "port": 1433,
      "database": "SalesDB"
    }
  }
}
```

The Parquet files written through ADBC differ from the native ones:

* `datetime2` keeps microseconds (the native driver writes milliseconds).
* `datetimeoffset` is a UTC timestamp (the native driver writes text such as `1/1/1970 12:00:00 AM +00:00`).

The ADBC driver needs its own license. It is included in the license file that arpe.io sends for LakeXpress, and LakeXpress passes it to FastBCP. If your license file has no ADBC block, provide the driver license with `ARPEIO_ADBC_LICENCE` (license text), `ARPEIO_ADBC_LICENCE_FILE` (path), or `arpeio.adbc.license_file=<path>` in `info.fastbcp_connection_string` (the connection string LakeXpress builds from `connection_string` does not carry it). Without any of these, the sync stops before the export with exit code 13 (`LICENSE_ERROR`). See [License Information](/latest/installation.md#license-information).

ADBC drivers ship with FastBCP for linux-x64 and win-x64. See [ADBC Extraction](/latest/configuration/database-configuration.md#adbc-extraction) for the other sources.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Login failed for user:** Confirm SQL Server authentication is enabled (not Windows-only), verify user exists with correct permissions, check password and IP restrictions.

**Cannot open database requested by the login:** Verify database name, check user access, confirm database is online: `SELECT state_desc FROM sys.databases WHERE name='YourDB'`

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# Teradata

Teradata is supported as a **source database** -- the database LakeXpress extracts data from. It cannot be used as the LakeXpress DB.

Connecting to Teradata requires the `teradatasql` Python driver.

## Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters")

| Parameter   | Required     | Description                                                                                                                                                                          |
| ----------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ds_type`   | Yes          | Must be `"teradata"`                                                                                                                                                                 |
| `auth_mode` | Yes          | `"classic"` for individual fields; `"connection_string"` to supply a full connection string                                                                                          |
| `server`    | No           | Hostname or IP (`host` is also accepted; default: `localhost`)                                                                                                                       |
| `port`      | No           | Port (default: 1025)                                                                                                                                                                 |
| `database`  | No           | Default database to connect to. If omitted, LakeXpress connects without selecting a default database                                                                                 |
| `username`  | Classic only | Teradata username                                                                                                                                                                    |
| `password`  | Classic only | Teradata password                                                                                                                                                                    |
| `use_adbc`  | No           | Export through FastBCP's ADBC driver (default: `true`). `false` uses FastBCP's native driver. Set at the same level as `ds_type`, not inside `info`. See [ADBC Export](#adbc-export) |

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

```
-- Source database user:
GRANT SELECT ON your_database TO lakexpress_user;

-- Metadata access (table discovery, column metadata, primary keys, foreign keys):
GRANT SELECT ON DBC.TablesV TO lakexpress_user;
GRANT SELECT ON DBC.ColumnsV TO lakexpress_user;
GRANT SELECT ON DBC.IndicesV TO lakexpress_user;
GRANT SELECT ON DBC.All_RI_ChildrenV TO lakexpress_user;
```

## Example: Source Database[​](#example-source-database "Direct link to Example: Source Database")

```
{
  "source_teradata": {
    "ds_type": "teradata",
    "auth_mode": "classic",
    "info": {
      "server": "teradata-prod.company.com",
      "port": 1025,
      "database": "DW",
      "username": "$env{LX_TD_USER}",
      "password": "$env{LX_TD_PASSWORD}"
    }
  }
}
```

## Example: Connection String Mode[​](#example-connection-string-mode "Direct link to Example: Connection String Mode")

Use `auth_mode: "connection_string"` when you need to pass advanced driver options.

```
{
  "source_teradata_cs": {
    "ds_type": "teradata",
    "auth_mode": "connection_string",
    "info": {
      "connection_string": "{\"host\":\"$env{LX_TD_HOST}\",\"dbs_port\":\"1025\",\"user\":\"$env{LX_TD_USER}\",\"password\":\"$env{LX_TD_PASSWORD}\",\"database\":\"DW\"}"
    }
  }
}
```

LakeXpress converts this JSON string for FastBCP's Teradata .NET driver: `host`, `dbs_port`, `user`, `password` and `database` are passed. Other keys (for example `logmech`) are not passed to FastBCP, and LakeXpress logs a warning. To pass them, add `fastbcp_connection_string` to `info`, in the .NET format (`Data Source=...;Port Number=1025;User Id=...;Password=...`), which FastBCP then uses as is.

## FastBCP Methods[​](#fastbcp-methods "Direct link to FastBCP Methods")

All generic FastBCP distribution methods are supported with Teradata sources:

| Method          | Description                                                                                                           |
| --------------- | --------------------------------------------------------------------------------------------------------------------- |
| `None`          | Single-threaded export (default)                                                                                      |
| `Random`        | Parallel export using an integer column                                                                               |
| `DataDriven`    | Split by column values into separate files                                                                            |
| `RangeId`       | Split by column min/max ranges                                                                                        |
| `Ntile`         | Evenly distributed chunks                                                                                             |
| `Timepartition` | Partition by date/time column -- supports `(col,year)`, `(col,year,month)`, `(col,year,month,day)`, `(col,year,week)` |

Timepartition splits exports into Hive-style paths (e.g., `year=2008/month=01/table.parquet`) where each parallel thread handles a specific time period. It is well suited for large historical tables with a date column.

```
# Single table
--fastbcp_table_config "ORDERS:Timepartition:(ORDER_DATE,year,month):8"

# Multiple tables with different date columns
--fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4" \
--fastbcp_table_config "PRESCRIPTION_DRUG_EVENTS:Timepartition:(SRVC_DT,year,month):4"
```

## ADBC Export[​](#adbc-export "Direct link to ADBC Export")

By default, FastBCP reads the source through the arpe.io ADBC driver for Teradata (connection type `adbc_teradata`). Set `"use_adbc": false` to use FastBCP's native .NET driver instead. The setting only changes the export: LakeXpress still discovers schemas and tables with `teradatasql`. The run log shows the path once per run, e.g. `Source extraction: adbc_teradata (ADBC)`.

```
{
  "teradata_native": {
    "ds_type": "teradata",
    "auth_mode": "classic",
    "use_adbc": false,
    "info": {
      "username": "$env{LX_TD_USER}",
      "password": "$env{LX_TD_PASSWORD}",
      "server": "teradata-prod.company.com",
      "port": 1025
    }
  }
}
```

How the driver logs on:

* By default it uses TLS on the HTTPS port (443) and needs no Teradata client software.
* If the server does not accept TLS, it falls back to the legacy port (1025, or `port`) through the Teradata Tools and Utilities (TTU) client, on Linux and Windows. The driver looks for TTU in `ARPETERADATA_TTU_PATH`, then `/opt/teradata/client`, then the system library path.
* With `auth_mode: "connection_string"`, the `sslmode`, `https_port`, `sslca` and `sslcapath` keys are passed to the ADBC driver. `"sslmode": "DISABLE"` always uses the legacy port. The native driver (`"use_adbc": false`) ignores these keys.

The ADBC driver needs its own license. It is included in the license file that arpe.io sends for LakeXpress, and LakeXpress passes it to FastBCP. If your license file has no ADBC block, provide the driver license with `ARPEIO_ADBC_LICENCE` (license text) or `ARPEIO_ADBC_LICENCE_FILE` (path). Without either, the sync stops before the export with exit code 13 (`LICENSE_ERROR`). A license file issued before 2026-10-26 has no FastBCP block: FastBCP then uses its own `FastBCP.lic`, which must carry the ADBC block. See [License Information](/latest/installation.md#license-information).

ADBC drivers ship with FastBCP for linux-x64 and win-x64. See [ADBC Extraction](/latest/configuration/database-configuration.md#adbc-extraction) for the other sources.

## See Also[​](#see-also "Direct link to See Also")

* [Database Connections Overview](/latest/configuration/database-configuration.md)Credentials file format, security best practices


---

# Azure Storage

Supports Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage via Service Principal authentication.

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "azure_01": {
    "ds_type": "azure",
    "auth_mode": "service_principal",
    "info": {
      "directory": "abfss://your-container@your-storage-account.dfs.core.windows.net/path/to/exports",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}
```

note

Use `$env{VAR_NAME}` syntax to reference environment variables. Set the variable before running LakeXpress (e.g., `export LX_AZURE_CLIENT_SECRET="your-secret"`).

| Field                 | Required | Description                                                     |
| --------------------- | -------- | --------------------------------------------------------------- |
| `ds_type`             | Yes      | Must be `"azure"`                                               |
| `auth_mode`           | Yes      | Must be `"service_principal"`                                   |
| `directory`           | Yes      | Azure storage URL (`abfss://` for ADLS Gen2, `abs://` for Blob) |
| `azure_client_id`     | Yes      | Application (client) ID from Azure AD app registration          |
| `azure_tenant_id`     | Yes      | Directory (tenant) ID from Azure AD                             |
| `azure_client_secret` | Yes      | Client secret value from Azure AD app                           |

## Storage URL Formats[​](#storage-url-formats "Direct link to Storage URL Formats")

**ADLS Gen2:**

```
abfss://container-name@storage-account-name.dfs.core.windows.net/path/to/exports
```

**Azure Blob Storage:**

```
abs://container-name@storage-account-name.blob.core.windows.net/path/to/exports
```

The container name goes before the `@`, and the storage account name goes after it (immediately before `.dfs.core.windows.net` / `.blob.core.windows.net`). This matches the `directory` value in the JSON example above.

## Azure Authentication Setup[​](#azure-authentication-setup "Direct link to Azure Authentication Setup")

**1. Create an Azure AD Application (Service Principal):**

In [Azure Portal App Registrations](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps):

* Click "New registration"
* Note the **Application (client) ID** and **Directory (tenant) ID**
* Go to "Certificates & secrets" -> "New client secret"
* Copy the **client secret value** (shown only once)

**2. Assign Storage Permissions:**

In your Storage Account -> Access Control (IAM), add the **"Storage Blob Data Contributor"** role to your service principal.

[Azure RBAC roles reference](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor)

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id azure_01
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

**ADLS Gen2:**

```
abfss://container@storage.dfs.core.windows.net/path/to/exports/schema_name/table_name/part-00000.parquet
```

**With `--sub_path`:**

```
abfss://container@storage.dfs.core.windows.net/path/to/exports/sub_path/schema_name/table_name/part-00000.parquet
```

## Required Permissions[​](#required-permissions "Direct link to Required Permissions")

The Service Principal needs **Storage Blob Data Contributor**, which grants:

* `Microsoft.Storage/storageAccounts/blobServices/containers/read`
* `Microsoft.Storage/storageAccounts/blobServices/containers/write`
* `Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action`

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem                        | Solution                                                                                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Authentication / access denied | Verify client ID, tenant ID, and secret. Confirm "Storage Blob Data Contributor" role. Check network access rules. Verify container exists. |
| Invalid URL format             | Use `abfss://` for ADLS Gen2, `abs://` for Blob. Format: `abfss://container@storageaccount.dfs.core.windows.net/path`.                      |
| Connection timeout             | Check firewall rules, storage account firewall allowlist, and private endpoint restrictions.                                                |


---

# Google Cloud Storage (GCS)

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "gcs_01": {
    "ds_type": "gcs",
    "auth_mode": "profile",
    "info": {
      "directory": "gs://your-bucket-name/path/to/exports",
      "profile": "/path/to/service-account-key.json"
    }
  }
}
```

| Field       | Required | Description                               |
| ----------- | -------- | ----------------------------------------- |
| `ds_type`   | Yes      | Must be `"gcs"`                           |
| `auth_mode` | Yes      | Must be `"profile"`                       |
| `directory` | Yes      | Full GCS URL (e.g., `gs://bucket/path`)   |
| `profile`   | Yes      | Path to GCS service account JSON key file |

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id gcs_01
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
gs://bucket-name/base_path/schema_name/table_name/part-00000.parquet
```

## Service Account Permissions[​](#service-account-permissions "Direct link to Service Account Permissions")

Required roles:

* `Storage Object Creator`
* `Storage Object Viewer` (if reading is needed)

Or custom IAM permissions:

* `storage.objects.create`
* `storage.objects.delete` (if overwriting)
* `storage.buckets.get`

## Creating a Service Account[​](#creating-a-service-account "Direct link to Creating a Service Account")

```
gcloud iam service-accounts create lakexpress-export \
    --display-name="LakeXpress Export Service Account"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectCreator"

gcloud iam service-accounts keys create service-account-key.json \
    --iam-account=lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem               | Solution                                                                                      |
| --------------------- | --------------------------------------------------------------------------------------------- |
| Authentication errors | Verify key file path. Check `Storage Object Creator` role. Ensure service account is enabled. |
| Quota exceeded        | Check GCS quotas in Google Cloud Console. Request increase if needed.                         |


---

# Local Filesystem

## Configuration[​](#configuration "Direct link to Configuration")

```
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --output_dir /data/exports
```

Relative paths

A relative `--output_dir` (for example `./exports`) is resolved against the directory where you run `config create`, and the absolute path is saved in the sync configuration. This requires LakeXpress 0.4.9 or later: earlier versions resolved relative paths from the FastBCP engine directory, so use an absolute path with them.

## Directory Structure[​](#directory-structure "Direct link to Directory Structure")

```
output_dir/
└── schema_name/
    └── table_name/
        ├── part-00000.parquet
        ├── part-00001.parquet
        └── ...
```

With `--sub_path staging/temp`:

```
exports/
└── staging/
    └── temp/
        └── schema_name/
            └── table_name/
                ├── part-00000.parquet
                └── ...
```

## Permissions[​](#permissions "Direct link to Permissions")

The user running LakeXpress needs write access to the output directory, sufficient disk space, and the ability to create subdirectories.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem           | Solution                                                          |
| ----------------- | ----------------------------------------------------------------- |
| Permission denied | `chmod 755 /path/to/output_dir`. Ensure parent directories exist. |


---

# Microsoft Fabric OneLake

Exports Parquet files to OneLake via Service Principal authentication.

## Configuration[​](#configuration "Direct link to Configuration")

```
{
  "onelake_01": {
    "ds_type": "onelake",
    "auth_mode": "service_principal",
    "info": {
      "directory": "onelake://your-workspace-name/your-lakehouse-name/",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}
```

| Field                 | Required | Description                           |
| --------------------- | -------- | ------------------------------------- |
| `ds_type`             | Yes      | Must be `"onelake"`                   |
| `auth_mode`           | Yes      | Must be `"service_principal"`         |
| `directory`           | Yes      | `onelake://workspace/lakehouse/path/` |
| `azure_client_id`     | Yes      | Application (client) ID from Azure AD |
| `azure_tenant_id`     | Yes      | Directory (tenant) ID from Azure AD   |
| `azure_client_secret` | Yes      | Client secret value from Azure AD app |

## URL Format[​](#url-format "Direct link to URL Format")

```
onelake://workspace-name/lakehouse-name/path/to/exports/
```

Example:

```
onelake://FABRICPOC/lakexpress_lakehouse/exports/
```

## Usage[​](#usage "Direct link to Usage")

```
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id onelake_01
```

With config creation:

```
python lxpress.py config create \
       -a credentials.json \
       --lxdb_auth_id lxdb_mssql \
       --source_db_auth_id source_postgres \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id onelake_01 \
       --generate_metadata
```

caution

`--generate_metadata` with a OneLake target is not fully reliable in the current release: schema discovery does not recognize `onelake` as its own storage type for metadata purposes, so it falls back to local-filesystem handling when building metadata paths. This can produce incorrect paths in the generated CDM metadata files. If you rely on `--generate_metadata`, verify the output paths carefully when the target storage is OneLake, or omit the flag until this is addressed.

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
onelake://workspace/lakehouse/base_path/schema_name/table_name/part-00000.parquet
```

With `--sub_path`:

```
onelake://workspace/lakehouse/base_path/sub_path/schema_name/table_name/part-00000.parquet
```

## Authentication Setup[​](#authentication-setup "Direct link to Authentication Setup")

**1. Create an Azure AD Application (Service Principal):**

In [Azure Portal App Registrations](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps):

* Click "New registration"
* Note the **Application (client) ID** and **Directory (tenant) ID**
* Go to "Certificates & secrets" -> "New client secret"
* Copy the **client secret value** (shown only once)

**2. Grant OneLake Access in Microsoft Fabric:**

In the Fabric portal, go to Workspace settings -> "Manage access" and add the service principal with Contributor or Admin permissions. For finer control, grant access at the Lakehouse level.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Problem                        | Solution                                                                                                     |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| Authentication / access denied | Verify client ID, tenant ID, and secret. Confirm Contributor role on workspace. Verify OneLake is enabled.   |
| Invalid URL format             | Use `onelake://` prefix. Format: `onelake://workspace-name/lakehouse-name/path/`. Ensure path ends with `/`. |
| Connection timeout             | Check firewall rules and verify Fabric workspace is network-accessible.                                      |


---

# S3-Compatible Storage

All S3-compatible providers use `ds_type: "s3"`. Provider differentiation is handled through `endpoint_url` in the AWS profile.

Supported providers: Amazon S3, OVH S3, MinIO, Wasabi, DigitalOcean Spaces, Backblaze B2, and any S3-compatible object storage.

## Configuration[​](#configuration "Direct link to Configuration")

* AWS S3 (recommended)
* S3-compatible (OVH, MinIO...)

Uses `~/.aws/config` and `~/.aws/credentials`:

```
{
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-bucket-name/lakexpress/exports",
      "profile": "your-aws-profile"
    }
  }
}
```

**`~/.aws/config`:**

```
[profile your-aws-profile]
region = us-east-1
```

**`~/.aws/credentials`:**

```
[your-aws-profile]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
```

| Field       | Required | Description                                |
| ----------- | -------- | ------------------------------------------ |
| `ds_type`   | Yes      | Must be `"s3"`                             |
| `auth_mode` | Yes      | Must be `"profile"`                        |
| `directory` | Yes      | Full S3 URL (e.g., `s3://bucket/path`)     |
| `profile`   | Yes      | AWS profile name from `~/.aws/credentials` |

***

Use an AWS profile with a custom `endpoint_url`.

**OVH S3:**

```
{
  "s3_02": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-ovh-bucket/lakexpress",
      "profile": "ovh"
    }
  }
}
```

```
[profile ovh]
endpoint_url = https://s3.gra.io.cloud.ovh.net
region = gra
```

**MinIO:**

```
{
  "s3_03": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://data-lake/exports",
      "profile": "minio"
    }
  }
}
```

```
[profile minio]
endpoint_url = http://localhost:9000
region = us-east-1
```

| Field       | Required | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"s3"`                                            |
| `auth_mode` | Yes      | Must be `"profile"`                                       |
| `directory` | Yes      | Full S3 URL (e.g., `s3://bucket/path`)                    |
| `profile`   | Yes      | AWS profile name (with `endpoint_url` in `~/.aws/config`) |

***

## Usage[​](#usage "Direct link to Usage")

```
# AWS S3
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id s3_01

# OVH S3
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id s3_02

# MinIO
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql \
        --source_db_auth_id source_postgres \
        --target_storage_id s3_03
```

## Path Structure[​](#path-structure "Direct link to Path Structure")

```
s3://bucket-name/base_path/schema_name/table_name/part-00000.parquet
```

With `--sub_path`:

```
s3://bucket-name/base_path/sub_path/schema_name/table_name/part-00000.parquet
```

## Required IAM Permissions (AWS S3)[​](#required-iam-permissions-aws-s3 "Direct link to Required IAM Permissions (AWS S3)")

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:s3:::your-bucket-name"
      ]
    }
  ]
}
```

## Backward Compatibility[​](#backward-compatibility "Direct link to Backward Compatibility")

Custom storage types ending in `_s3` (e.g., `minio_s3`, `custom_s3`) still work, but `ds_type: "s3"` is preferred.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

* AWS S3 (recommended)
* S3-compatible (OVH, MinIO...)

| Problem                           | Solution                                                                                                                   |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Access denied / credential errors | Verify credentials: `aws s3 ls s3://your-bucket --profile your-profile`. Check IAM includes `s3:PutObject`. Verify region. |
| Slow uploads                      | Enable S3 Transfer Acceleration. Use S3 in the same region as your source database.                                        |

| Problem           | Solution                                                                                            |
| ----------------- | --------------------------------------------------------------------------------------------------- |
| Connection errors | Verify endpoint URL. Check firewall rules. Ensure valid SSL certificates. Verify credential format. |


---

# Intermediate Storage

After extracting data from a source database, LakeXpress stages the results as Parquet files in intermediate storage before publishing to a target platform (such as Snowflake or Databricks). This staging step decouples extraction from publishing: you can extract once and publish to multiple targets, re-publish without re-extracting, or inspect the raw files before they reach the destination.

LakeXpress supports two categories of intermediate storage:

| Option               | Flag                  | Description                                            |
| -------------------- | --------------------- | ------------------------------------------------------ |
| **Local Filesystem** | `--output_dir`        | Stage files in a local directory                       |
| **Cloud Storage**    | `--target_storage_id` | Stage files in S3, GCS, Azure Blob Storage, or OneLake |

At least one of `--output_dir` or `--target_storage_id` is required per pipeline run; omitting both raises an error. They are not mutually exclusive at the CLI level, but in practice you normally set only one: local filesystem exports use `--output_dir`, and cloud storage exports use `--target_storage_id`.

note

Cloud storage (S3, GCS, Azure, OneLake) is a staging location, not a publishing target. Publishing targets such as Snowflake, Databricks, and BigQuery are configured separately with `--publish_target`. See the [Snowflake Publishing Guide](/latest/integrations/snowflake-publishing.md) for an example of how publishing follows storage.

## Sub-Path Option[​](#sub-path-option "Direct link to Sub-Path Option")

All backends support `--sub_path` to insert an intermediate directory level:

```
./LakeXpress --target_storage_id s3_01 --sub_path staging/daily/2025-01-15 ...
```

Produces:

```
s3://bucket/base_path/staging/daily/2025-01-15/schema_name/table_name/*.parquet
```

Use cases: date partitioning (`staging/2025/01/15`), environment separation (`dev/exports` vs `prod/exports`), project organization (`project-a/datasets`).

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/latest/guides/quickstart.md)
* [CLI Reference](/latest/cli/.md)
* [Database Configuration](/latest/configuration/database-configuration.md)
* [Examples & Recipes](/latest/guides/command-cookbook.md)


---

# Examples & Recipes

Tested commands for common LakeXpress scenarios, grouped by task.

Commands use Linux syntax. On Windows PowerShell, replace `\` with `` ` `` at the end of lines, and `./LakeXpress` with `.\LakeXpress.exe`.

The commands leave out `--fastbcp_dir_path`: LakeXpress finds its bundled FastBCP engine automatically. To use another FastBCP installation, add `--fastbcp_dir_path <folder containing FastBCP>` to `config create`.

The credentials entry names (`lxdb_postgres`, `source_postgres`, `s3_01`, `snowflake_prod`...) are examples: use the names from your own `credentials.json`.

License

The examples assume the license file is saved in a default location (`~/.lakexpress/license.lic`, `%APPDATA%\LakeXpress\license.lic` or `./license.lic`). Otherwise add `--license_file <path>` to every command, after the command name:

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID> \
  --license_file ~/licenses/LakeXpress.lic
```

For scheduled jobs, CI and containers, see [License Resolution](/latest/cli/.md#license-resolution).

Required sync arguments

The `sync`, `sync[export]`, and `sync[publish]` commands require `-a`, `--lxdb_auth_id`, and `--sync_id`. The `<SYNC_ID>` in the examples refers to the sync ID printed by `config create`.

## Manage LakeXpress[​](#manage-lakexpress "Direct link to Manage LakeXpress")

* [LakeXpress DB](/latest/guides/recipes/lakexpress-db.md)initialize, empty, drop and unlock the LakeXpress DB, for each supported backend
* [Configuration Management](/latest/guides/recipes/configuration-management.md)create, list and delete sync configurations, check their status

## Export[​](#export "Direct link to Export")

* [![](/img/logos/local-files.svg)Local Exports](/latest/guides/recipes/local-exports.md)PostgreSQL, SQL Server, MySQL, MariaDB and Teradata to local disk
* [![](/img/logos/aws-s3.png)AWS S3](/latest/guides/recipes/s3.md)Parquet files on S3 and S3-compatible storage
* [![](/img/logos/azure-blob.svg)Azure Storage](/latest/guides/recipes/azure.md)Parquet files on Azure Blob Storage and ADLS Gen2
* [![](/img/logos/gcs.png)Google Cloud Storage](/latest/guides/recipes/gcs.md)Parquet files on GCS
* [![](/img/logos/onelake.png)OneLake](/latest/guides/recipes/onelake.md)Parquet files on Microsoft OneLake

## Publish[​](#publish "Direct link to Publish")

* [![](/img/logos/snowflake.svg)Snowflake](/latest/guides/recipes/snowflake.md)external and internal tables
* [![](/img/logos/databricks.svg)Databricks](/latest/guides/recipes/databricks.md)tables in Unity Catalog
* [![](/img/logos/aws-glue.svg)AWS Glue](/latest/guides/recipes/glue.md)tables in the Glue Data Catalog, from S3
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric](/latest/guides/recipes/fabric.md)tables in Fabric, from OneLake
* [![](/img/logos/bigquery.svg)BigQuery](/latest/guides/recipes/bigquery.md)external and internal tables
* [Amazon Redshift](/latest/guides/recipes/redshift.md)external and internal tables
* [![](/img/logos/motherduck.svg)MotherDuck](/latest/guides/recipes/motherduck.md)external and internal tables
* [![](/img/logos/ducklake.svg)DuckLake](/latest/guides/recipes/ducklake.md)external tables

## Refine and operate[​](#refine-and-operate "Direct link to Refine and operate")

* [Table Filtering](/latest/guides/recipes/table-filtering.md)include and exclude table patterns
* [Incremental Sync](/latest/guides/recipes/incremental-sync.md)export only new or changed rows
* [Resume Failed Exports](/latest/guides/recipes/resume-failed-exports.md)retry only the tables that failed

## See Also[​](#see-also "Direct link to See Also")

* [Quick Start Guide](/latest/guides/quickstart.md)
* [CLI Reference](/latest/cli/.md)
* [Incremental Sync](/latest/advanced/incremental-sync.md)
* [![](/img/logos/snowflake.svg)Snowflake Publishing](/latest/integrations/snowflake-publishing.md)
* [![](/img/logos/databricks.svg)Databricks Publishing](/latest/integrations/databricks-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue Publishing](/latest/integrations/glue-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md)


---

# Quick start guide

This guide takes you from zero to a first export: every table of a PostgreSQL schema written as Parquet files to a local directory.

## 1. Install LakeXpress[​](#1-install-lakexpress "Direct link to 1. Install LakeXpress")

Download the LakeXpress archive for your platform and unzip it. It bundles the FastBCP engine, and your license file covers both. Keep the license file you received by email: every command needs it. See the [Installation guide](/latest/installation.md) for details.

## 2. Prepare the databases[​](#2-prepare-the-databases "Direct link to 2. Prepare the databases")

LakeXpress connects to two databases: the **source database** you export from, and the **LakeXpress DB**, where it stores its configuration and sync history.

### Source database: a read-only user[​](#source-database-a-read-only-user "Direct link to Source database: a read-only user")

Create a user with `SELECT` privileges on the schema you want to export:

```
CREATE USER lakexpress_reader WITH PASSWORD 'your_password';
GRANT USAGE ON SCHEMA public TO lakexpress_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO lakexpress_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT ON TABLES TO lakexpress_reader;
```

For other source databases, see [Database Connections](/latest/configuration/database-configuration.md).

### LakeXpress DB: an empty database[​](#lakexpress-db-an-empty-database "Direct link to LakeXpress DB: an empty database")

Create an empty database and a user that owns it. LakeXpress creates its own tables inside it.

```
CREATE USER lakexpress_admin WITH PASSWORD 'your_password';
CREATE DATABASE lakexpress_log OWNER lakexpress_admin;
```

Quick testing

SQLite or DuckDB can also be used as the LakeXpress DB. They need no server: point to a file path in the credentials file. For these and the other backends (SQL Server, MySQL), see [Set up the LakeXpress DB](/latest/lakexpress-db/.md).

## 3. Create the credentials file[​](#3-create-the-credentials-file "Direct link to 3. Create the credentials file")

All connections are described in one JSON file. Save this as `credentials.json`:

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  }
}
```

The top-level keys (`lxdb_postgres`, `source_postgres`) are the IDs you pass to the commands below.

`$env{VAR_NAME}` reads a value from an environment variable (an error is raised if it is not set). Plain-text values also work. Set the variables before running LakeXpress:

* Windows (PowerShell)
* Linux

```
$env:LX_LXDB_USER = "lakexpress_admin"
$env:LX_LXDB_PASSWORD = "your_password"
$env:LX_PG_USER = "lakexpress_reader"
$env:LX_PG_PASSWORD = "your_password"
```

```
export LX_LXDB_USER="lakexpress_admin"
export LX_LXDB_PASSWORD="your_password"
export LX_PG_USER="lakexpress_reader"
export LX_PG_PASSWORD="your_password"
```

## 4. Initialize the LakeXpress DB[​](#4-initialize-the-lakexpress-db "Direct link to 4. Initialize the LakeXpress DB")

Create the LakeXpress tables and check that the connection works:

* Windows (PowerShell)
* Linux

```
.\LakeXpress.exe lxdb init `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --license_file C:\licenses\LakeXpress.lic
```

```
./LakeXpress lxdb init \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --license_file ~/licenses/LakeXpress.lic
```

This step is optional -- the first sync creates the tables if needed -- but it catches connection problems early.

`--license_file` points to your license file. Every command accepts it, after the command name. You can leave it out when the file is saved in a default location: `~/.lakexpress/license.lic` (Linux), `%APPDATA%\LakeXpress\license.lic` (Windows) or `./license.lic`. See [License Resolution](/latest/cli/.md#license-resolution) for the other ways to provide it.

## 5. Create a sync configuration[​](#5-create-a-sync-configuration "Direct link to 5. Create a sync configuration")

A sync configuration says what to export and where. It is stored in the LakeXpress DB and reused by every sync.

* Windows (PowerShell)
* Linux

```
.\LakeXpress.exe config create `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --source_db_auth_id source_postgres `
  --source_schema_name public `
  --output_dir .\exports `
  --n_jobs 4 `
  --fastbcp_p 2 `
  --license_file C:\licenses\LakeXpress.lic
```

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_schema_name public \
  --output_dir ./exports \
  --n_jobs 4 \
  --fastbcp_p 2 \
  --license_file ~/licenses/LakeXpress.lic
```

This exports every table of the `public` schema of `production_db` (the database named in `source_postgres`), 4 tables in parallel, each table of at least 100,000 rows (`--large_table_threshold`) split into 2 partitions. The relative path `./exports` is resolved against the current directory and saved as an absolute path, so later syncs write to the same place wherever they are run from (LakeXpress 0.4.9 or later; with earlier versions, pass an absolute path). The `exports` directory is created if it does not exist, and the FastBCP engine bundled in the archive (`engine/`) is found automatically. If you want to use a FastBCP installed elsewhere, add `--fastbcp_dir_path <folder containing FastBCP>`.

The command ends by printing a **`sync_id`** and the exact `sync` command to run next.

## 6. Run the sync[​](#6-run-the-sync "Direct link to 6. Run the sync")

* Windows (PowerShell)
* Linux

```
.\LakeXpress.exe sync `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID> `
  --license_file C:\licenses\LakeXpress.lic
```

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID> \
  --license_file ~/licenses/LakeXpress.lic
```

Replace `<SYNC_ID>` with the `sync_id` printed by `config create`. LakeXpress shows progress as tables are exported.

## 7. Check the result[​](#7-check-the-result "Direct link to 7. Check the result")

Each table is written as Parquet files under `exports/public/<table_name>/`, in the directory where you ran `config create`. To see the status of the run:

* Windows (PowerShell)
* Linux

```
.\LakeXpress.exe status `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres `
  --sync_id <SYNC_ID>
```

```
./LakeXpress status \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID>
```

To export again later, run the same `sync` command: the configuration is reused.


---

# Azure Storage Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

**PostgreSQL to Azure**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id azure_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server to Azure**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworks2019 \
    --source_schema_name sales \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id azure_01 \
    --sub_path adventureworks/sales \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

**credentials.json (Azure with Service Principal):**

```
{
  "azure_01": {
    "ds_type": "azure",
    "auth_mode": "service_principal",
    "info": {
      "directory": "abfss://datalake.dfs.core.windows.net/exports",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  },
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "localhost",
      "port": 1433,
      "database": "adventureworks2019"
    }
  }
}
```


---

# BigQuery Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

The credentials entries used below, and the BigQuery setup they rely on, are described in [BigQuery Publishing](/latest/integrations/bigquery-publishing.md).

## Example 1: External Tables (Default)[​](#example-1-external-tables-default "Direct link to Example 1: External Tables (Default)")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target bigquery_prod
```

Creates dataset `EXT_public` (the default schema pattern for the default `external` publish method) with external tables pointing to GCS Parquet files. Pass `--publish_schema_pattern "{schema}"` explicitly if you want the dataset named `public` instead.

## Example 2: Native Tables with Parallel Execution[​](#example-2-native-tables-with-parallel-execution "Direct link to Example 2: Native Tables with Parallel Execution")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target bigquery_prod \
    --publish_method internal \
    --publish_schema_pattern "lx_{schema}" \
    --n_jobs 4
```

Creates dataset `lx_tpch_1` with native tables loaded from GCS.

## Example 3: Daily Snapshots[​](#example-3-daily-snapshots "Direct link to Example 3: Daily Snapshots")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name sales \
    --target_storage_id gcs_01 \
    --sub_path "daily/$(date +%Y%m%d)" \
    --publish_target bigquery_prod \
    --publish_schema_pattern "sales_{date}"
```

Creates dataset `sales_20251210`, data stored in `gs://bucket/lakexpress/daily/20251210/`.

## Example 4: EU Data Residency[​](#example-4-eu-data-residency "Direct link to Example 4: EU Data Residency")

```
{
    "bigquery_eu": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-eu-project",
            "location": "EU",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name gdpr_data \
    --target_storage_id gcs_eu \
    --sub_path mydata \
    --publish_target bigquery_eu
```

## Example 5: With Primary Key Constraints[​](#example-5-with-primary-key-constraints "Direct link to Example 5: With Primary Key Constraints")

Add `--pk_constraints` to give the native (`internal`) tables LakeXpress creates the source primary key, as `PRIMARY KEY (...) NOT ENFORCED`:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --publish_method internal \
    --publish_target bigquery_prod \
    --pk_constraints
```


---

# Configuration Management

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

```
# Create a sync configuration (the other recipes show all the options)
./LakeXpress config create -a credentials.json --lxdb_auth_id lxdb_mssql --source_db_auth_id source_postgres --source_schema_name public --output_dir ./exports

# List the sync configurations
./LakeXpress config list -a credentials.json --lxdb_auth_id lxdb_mssql

# Show the runs of a sync configuration and their status
./LakeXpress status -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID>
```

## Delete a Sync Configuration[​](#delete-a-sync-configuration "Direct link to Delete a Sync Configuration")

Without `--confirm`, `config delete` only shows what would be deleted: the configuration, its runs, table metadata and watermarks. Exported files and published tables are kept.

```
# Preview
./LakeXpress config delete -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID>

# Delete
./LakeXpress config delete -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID> --confirm
```

## Export Now, Publish Later[​](#export-now-publish-later "Direct link to Export Now, Publish Later")

`sync` exports and then publishes. The two steps can also be run separately, for example to publish outside business hours:

```
# Export only
./LakeXpress 'sync[export]' -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID>

# Publish the files of the latest completed run
./LakeXpress 'sync[publish]' -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID>
```

## Clean Up Stale Runs[​](#clean-up-stale-runs "Direct link to Clean Up Stale Runs")

Remove runs stuck in `running` after a crash, or `failed` runs, older than 7 days. Completed runs and exported files are kept:

```
# Preview
./LakeXpress cleanup -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID> --older-than 7d --dry-run

# Remove
./LakeXpress cleanup -a credentials.json --lxdb_auth_id lxdb_mssql --sync_id <SYNC_ID> --older-than 7d
```

## Legacy YAML Configuration[​](#legacy-yaml-configuration "Direct link to Legacy YAML Configuration")

Each `sync` run writes a copy of its configuration next to its log file. It can be run again with `run`:

```
./LakeXpress run -c lx_20251222_101500_<RUN_ID>.yml
```


---

# Databricks Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

Export to S3, then create Databricks Unity Catalog tables.

**PostgreSQL to Databricks (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_method external

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**PostgreSQL to Databricks (Internal/Managed Delta Tables)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server to Databricks (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_schema_pattern "adventureworksdw" \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## More Examples[​](#more-examples "Direct link to More Examples")

The credentials entries used below, and the Databricks setup they rely on, are described in [Databricks Publishing](/latest/integrations/databricks-publishing.md).

### Example 1: External Tables (Data in S3)[​](#example-1-external-tables-data-in-s3 "Direct link to Example 1: External Tables (Data in S3)")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_method external
```

Result: External Parquet tables in `lakexpress_catalog.EXT_public` reading from S3 (default schema pattern `EXT_{schema}` for external tables).

### Example 2: Managed Delta Tables[​](#example-2-managed-delta-tables "Direct link to Example 2: Managed Delta Tables")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_method internal \
    --n_jobs 4
```

Result: Delta tables in `lakexpress_catalog.tpch_1` loaded via COPY INTO.

### Example 3: Custom Schema Pattern with Primary Keys[​](#example-3-custom-schema-pattern-with-primary-keys "Direct link to Example 3: Custom Schema Pattern with Primary Keys")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name sales \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_schema_pattern "raw_{schema}" \
    --publish_method internal \
    --pk_constraints
```

Result: Managed Delta tables in `{catalog}.raw_sales` (`customer`, `orders`, etc.) with PRIMARY KEY constraints, where `{catalog}` is the value of the `catalog` field in the `databricks_prod` credential -- there is no `--databricks_catalog` override flag.

### Example 4: Parallel Table Creation[​](#example-4-parallel-table-creation "Direct link to Example 4: Parallel Table Creation")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --n_jobs 8
```


---

# DuckLake Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

The credentials entries used below, and the DuckLake setup they rely on, are described in [DuckLake Publishing](/latest/integrations/ducklake-publishing.md).

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id s3_01 \
  --sub_path mydata \
  --publish_target ducklake_catalog \
  --publish_schema_pattern "{schema}" \
  --publish_table_pattern "{table}"
```

`--publish_target`

Credential ID of the DuckLake catalog entry. See [Catalog Configuration](/latest/integrations/ducklake-publishing.md#catalog-configuration).

`--target_storage_id`or`--output_dir`

Where the exported Parquet files are written: a storage entry, or `--output_dir` for a local export.

On cloud storage (S3, GCS, Azure), LakeXpress generates a `CREATE SECRET` statement so DuckDB can read the files.

`--publish_schema_pattern`/`--publish_table_pattern`

Same tokens as the other publishers: `{schema}`, `{table}`, `{date}`, `{timestamp}`, `{uuid}`, `{subpath}`. `{database}` resolves to an empty string.

* Schema default: `EXT_{schema}`, or `{schema}` with `--publish_method internal`
* Table default: `{table}`

`--publish_method`No effect on DuckLake

DuckLake registers the exported Parquet files by reference, without copying them. Only tables with the `upsert` or `delete_insert` strategy are copied into a DuckLake-managed table.

It still sets the default schema pattern when `--publish_schema_pattern` is not given: `{schema}` for `internal`, `EXT_{schema}` for `external` (the default).


---

# Microsoft Fabric Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

Export to OneLake, then create Fabric Lakehouse tables.

**PostgreSQL to Fabric (Internal/Delta Tables)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**PostgreSQL to Fabric (External/SQL Views)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method external \
    --publish_table_pattern "VW_{schema}_{table}" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## More Examples[​](#more-examples "Direct link to More Examples")

The credentials entries used below, and the Microsoft Fabric setup they rely on, are described in [Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md).

### Example 1: PostgreSQL to Fabric Delta Tables[​](#example-1-postgresql-to-fabric-delta-tables "Direct link to Example 1: PostgreSQL to Fabric Delta Tables")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Tables (`tpch_1_customer`, `tpch_1_orders`, `tpch_1_lineitem`, etc., from the default pattern `{schema}_{table}`) appear in the Lakehouse Tables section, queryable via SQL analytics endpoint and Power BI.

### Example 2: SQL Server to Fabric with Custom Naming[​](#example-2-sql-server-to-fabric-with-custom-naming "Direct link to Example 2: SQL Server to Fabric with Custom Naming")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --publish_table_pattern "dw_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Creates tables `dw_DimCustomer`, `dw_DimProduct`, `dw_FactSales`, etc.

### Example 3: SQL Analytics Views (External)[​](#example-3-sql-analytics-views-external "Direct link to Example 3: SQL Analytics Views (External)")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method external \
    --publish_table_pattern "VW_{schema}_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Parquet files land in the Files section; views `VW_tpch_1_customer`, `VW_tpch_1_orders`, etc. appear in the SQL analytics endpoint.

### Example 4: Parallel Table Creation[​](#example-4-parallel-table-creation "Direct link to Example 4: Parallel Table Creation")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 4 \
    --n_jobs 8 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

### Example 5: MySQL to Fabric[​](#example-5-mysql-to-fabric "Direct link to Example 5: MySQL to Fabric")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_mysql \
    --source_db_name ecommerce \
    --source_schema_name ecommerce \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_01 \
    --publish_target fabric_lakehouse \
    --publish_method internal \
    --publish_table_pattern "mysql_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```


---

# Google Cloud Storage Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

**PostgreSQL to GCS**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id gcs_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```


---

# AWS Glue Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

Export to S3, then register tables in AWS Glue Data Catalog.

**PostgreSQL to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**PostgreSQL to AWS Glue with Custom FastBCP Config**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:TO_CHAR(l_shipdate, 'YYYY'):8;orders:Ctid::4" \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server to AWS Glue with Custom FastBCP Config**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Physloc::4" \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_database_name "lakexpress_tpch" \
    --publish_table_pattern "{schema}_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server AdventureWorks to AWS Glue**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target glue_catalog \
    --publish_schema_pattern "adventureworksdw"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## More Examples[​](#more-examples "Direct link to More Examples")

The credentials entries used below, and the AWS Glue setup they rely on, are described in [AWS Glue Publishing](/latest/integrations/glue-publishing.md).

### Example 1: Basic Export to Glue[​](#example-1-basic-export-to-glue "Direct link to Example 1: Basic Export to Glue")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --publish_target glue_catalog
```

Creates database `ext_public` (Glue lowercases database names) with all tables from the `public` schema.

### Example 2: Custom Naming with Parallel Execution[​](#example-2-custom-naming-with-parallel-execution "Direct link to Example 2: Custom Naming with Parallel Execution")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lx_{schema}" \
    --publish_table_pattern "{table}" \
    --n_jobs 4
```

Creates database `lx_tpch_1` with tables built in parallel.

### Example 3: Literal Database Name[​](#example-3-literal-database-name "Direct link to Example 3: Literal Database Name")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_database_name analytics
```

Creates (or reuses) the Glue database named exactly `analytics`, ignoring `--publish_schema_pattern`.

### Example 4: Date-Based Snapshots[​](#example-4-date-based-snapshots "Direct link to Example 4: Date-Based Snapshots")

```
./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name sales \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_schema_pattern "sales_{date}" \
    --sub_path "daily/$(date +%Y%m%d)"
```

Creates database `sales_20251210`, data stored at `s3://bucket/lakexpress/daily/20251210/`.


---

# Incremental Sync

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

Each sync exports only the rows added since the previous one, tracked with a watermark column per table. How it works: [Incremental Sync](/latest/advanced/incremental-sync.md).

## Step by Step[​](#step-by-step "Direct link to Step by Step")

### Step 1: Create Configuration with Incremental Tables[​](#step-1-create-configuration-with-incremental-tables "Direct link to Step 1: Create Configuration with Incremental Tables")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --source_db_auth_id source_postgres \
  --source_db_name tpch \
  --source_schema_name tpch_1_incremental \
  --target_storage_id s3_01 \
  --incremental_table "tpch_1_incremental.orders:o_orderdate:date" \
  --incremental_table "tpch_1_incremental.lineitem:l_shipdate:date" \
  --generate_metadata \
  --n_jobs 4 \
  --fastbcp_p 2
```

| Parameter                                                           | Meaning                                   |
| ------------------------------------------------------------------- | ----------------------------------------- |
| `--incremental_table "tpch_1_incremental.orders:o_orderdate:date"`  | Track orders by `o_orderdate` column      |
| `--incremental_table "tpch_1_incremental.lineitem:l_shipdate:date"` | Track lineitem by `l_shipdate` column     |
| `--generate_metadata`                                               | Generate CDM metadata for exported tables |

### Step 2: Run First Sync[​](#step-2-run-first-sync "Direct link to Step 2: Run First Sync")

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --sync_id <SYNC_ID>
```

First sync behavior:

* Exports **all** rows up to the upper bound (`o_orderdate <= d_new` / `l_shipdate <= d_new`), where `d_new` is `MAX(column)` in the source at run start
* Records `d_new` as each table's `last_exported_value` watermark
* Non-incremental tables are fully exported
* Stores watermarks in the LakeXpress DB

### Step 3: Subsequent Syncs[​](#step-3-subsequent-syncs "Direct link to Step 3: Subsequent Syncs")

```
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --sync_id <SYNC_ID>
```

Subsequent sync behavior:

* Loads previous watermarks (`d_old`) from the LakeXpress DB, unchanged
* Computes a new upper bound `d_new` = `MAX(column)` in the source at run start
* Exports rows where `o_orderdate > d_old AND o_orderdate <= d_new`
* Exports rows where `l_shipdate > d_old AND l_shipdate <= d_new`
* Updates each watermark's `last_exported_value` to the `d_new` used for that run
* Non-incremental tables are fully exported again

## Daily Order Processing[​](#daily-order-processing "Direct link to Daily Order Processing")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name ecommerce \
  --source_schema_name public \
  --target_storage_id s3_01 \
  --incremental_table "public.orders:created_at:datetime" \
  --incremental_table "public.order_items:created_at:datetime" \
  --publish_target snowflake_prod \
  --n_jobs 4

# Run daily via cron
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID>
```

* Day 1: Exports 1,000,000 orders (full load)
* Day 2: Exports \~5,000 new orders (incremental)
* Day 3: Exports \~4,800 new orders (incremental)
* Other tables (customers, products) fully exported daily

## Event Log Ingestion[​](#event-log-ingestion "Direct link to Event Log Ingestion")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --source_db_auth_id source_postgres \
  --source_db_name analytics \
  --source_schema_name events \
  --target_storage_id s3_01 \
  --incremental_table "events.pageviews:event_time:datetime" \
  --incremental_table "events.clicks:event_time:datetime" \
  --incremental_table "events.conversions:event_time:datetime" \
  --sub_path production/events \
  --n_jobs 8 \
  --fastbcp_p 4

# Run every 10 minutes
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --sync_id <SYNC_ID>
```

* Ingests events from multiple tables continuously

## Time-Series Metrics[​](#time-series-metrics "Direct link to Time-Series Metrics")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --source_db_auth_id source_postgres \
  --source_db_name monitoring \
  --source_schema_name metrics \
  --target_storage_id azure_01 \
  --incremental_table "metrics.cpu_usage:recorded_at:datetime" \
  --incremental_table "metrics.memory_usage:recorded_at:datetime" \
  --incremental_table "metrics.disk_io:recorded_at:datetime" \
  --n_jobs 4 \
  --generate_metadata

# Run every 5 minutes
./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_sqlite \
  --sync_id <SYNC_ID>
```

* High-frequency metric collection to Azure storage
* Each sync captures the last 5+ minutes of data

## Publish to Snowflake[​](#publish-to-snowflake "Direct link to Publish to Snowflake")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_schema_name sales \
  --incremental_table "sales.orders:order_date:date" \
  --incremental_table "sales.returns:return_date:date" \
  --target_storage_id s3_01 \
  --publish_target snowflake_prod \
  --publish_method internal \
  --publish_schema_pattern "{schema}_incremental" \
  --n_jobs 4

./LakeXpress sync \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --sync_id <SYNC_ID>
```

Creates Snowflake tables continuously updated with new data. Non-incremental tables are fully exported and published on each sync.

## Debug a Sync[​](#debug-a-sync "Direct link to Debug a Sync")

Run with `--log_level DEBUG` for detailed logs:

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID> \
    --log_level DEBUG
```

## See Also[​](#see-also "Direct link to See Also")

* [Incremental Sync](/latest/advanced/incremental-sync.md)configuration syntax and supported column types
* [Loading Strategies](/latest/advanced/incremental/loading-strategies.md)append, upsert and delete-insert
* [Watermarks](/latest/advanced/incremental/watermarks.md)watermark tracking, late-arriving data and safety lag


---

# LakeXpress DB Commands

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

`drop`, `truncate` and `release-locks` only show what they would do unless `--confirm` is given. Run them once without it to check.

```
# Create the LakeXpress tables (safe to run again)
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_mssql

# Drop the LakeXpress tables and their data
./LakeXpress lxdb drop -a credentials.json --lxdb_auth_id lxdb_mssql --confirm

# Delete all data (configurations, runs, watermarks), keep the tables
./LakeXpress lxdb truncate -a credentials.json --lxdb_auth_id lxdb_mssql --confirm
```

## Locks Left by a Crashed Sync[​](#locks-left-by-a-crashed-sync "Direct link to Locks Left by a Crashed Sync")

Incremental syncs lock the tables they export. If a sync crashes or is killed, its locks block the next runs:

```
# See the locked tables and their IDs
./LakeXpress lxdb locks -a credentials.json --lxdb_auth_id lxdb_mssql

# Release the locks older than 24 hours (the default age)
./LakeXpress lxdb release-locks -a credentials.json --lxdb_auth_id lxdb_mssql --confirm

# Release the locks older than 2 hours
./LakeXpress lxdb release-locks -a credentials.json --lxdb_auth_id lxdb_mssql --max_age_hours 2 --confirm

# Release the lock of one table, by the ID shown by 'lxdb locks'
./LakeXpress lxdb release-locks -a credentials.json --lxdb_auth_id lxdb_mssql --table_id 42 --confirm
```

## Different LakeXpress DB Types[​](#different-lakexpress-db-types "Direct link to Different LakeXpress DB Types")

```
# SQL Server as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_mssql

# PostgreSQL as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_postgres

# DuckDB as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_duckdb

# MySQL as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_mysql

# SQLite as LakeXpress DB
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_sqlite
```


---

# Local Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

## PostgreSQL Source[​](#postgresql-source "Direct link to PostgreSQL Source")

**PostgreSQL to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## SQL Server Source[​](#sql-server-source "Direct link to SQL Server Source")

**SQL Server to Local (PostgreSQL LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (SQLite LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_sqlite \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_sqlite \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (DuckDB LakeXpress DB)**

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_duckdb

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_duckdb \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_duckdb \
    --sync_id <SYNC_ID>
```

**SQL Server to Local (MySQL LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mysql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mysql \
    --sync_id <SYNC_ID>
```

## MySQL Source[​](#mysql-source "Direct link to MySQL Source")

**MySQL to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mysql \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## MariaDB Source[​](#mariadb-source "Direct link to MariaDB Source")

**MariaDB to Local (SQL Server LakeXpress DB)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mariadb \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --output_dir /tmp/tpch/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## Teradata Source[​](#teradata-source "Direct link to Teradata Source")

**Teradata to Local with Timepartition**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_teradata \
    --source_db_name DW \
    --source_schema_name claims \
    --fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4" \
    --fastbcp_table_config "PRESCRIPTION_DRUG_EVENTS:Timepartition:(SRVC_DT,year,month):4" \
    --n_jobs 4 \
    --output_dir /tmp/claims/ \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```


---

# MotherDuck Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

The credentials entries used below, and the MotherDuck setup they rely on, are described in [MotherDuck Publishing](/latest/integrations/motherduck-publishing.md).

## Example 1: Basic Export with External Views[​](#example-1-basic-export-with-external-views "Direct link to Example 1: Basic Export with External Views")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --sub_path mydata \
    --publish_target motherduck_prod
```

**Result:** Database `my_analytics`, schema `EXT_public` (the default schema pattern for the default `external` publish method), external views over S3 Parquet files. Pass `--publish_schema_pattern "{schema}"` explicitly for a schema named `public` instead.

## Example 2: Internal Tables with Parallel Execution[​](#example-2-internal-tables-with-parallel-execution "Direct link to Example 2: Internal Tables with Parallel Execution")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --sub_path mydata \
    --publish_target motherduck_prod \
    --publish_method internal \
    --publish_schema_pattern "lx_{schema}" \
    --n_jobs 4
```

**Result:** Database `my_analytics`, schema `lx_tpch_1`, native tables loaded from S3.

## Example 3: Daily Snapshots[​](#example-3-daily-snapshots "Direct link to Example 3: Daily Snapshots")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name sales \
    --target_storage_id s3_01 \
    --sub_path "daily/$(date +%Y%m%d)" \
    --publish_target motherduck_prod \
    --publish_schema_pattern "sales_{date}"
```

**Result:** Schema `sales_20251210`, data at `s3://bucket/lakexpress/daily/20251210/`.

## Example 4: Environment Variable Token[​](#example-4-environment-variable-token "Direct link to Example 4: Environment Variable Token")

```
export motherduck_token="your-token-here"

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --sub_path mydata \
    --publish_target motherduck_env
```

## Example 5: GCS Storage Backend[​](#example-5-gcs-storage-backend "Direct link to Example 5: GCS Storage Backend")

```
{
    "gcs_01": {
        "ds_type": "gcs",
        "auth_mode": "profile",
        "info": {
            "directory": "gs://my-datalake-bucket/lakexpress/",
            "profile": "/path/to/service-account.json"
        }
    },
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-token"
        }
    }
}
```

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target motherduck_prod
```

## Example 6: With Primary Key Constraints[​](#example-6-with-primary-key-constraints "Direct link to Example 6: With Primary Key Constraints")

Add `--pk_constraints` to include primary key definitions in native (`internal`) table DDL:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --publish_method internal \
    --publish_target motherduck_prod \
    --pk_constraints
```


---

# OneLake Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

**PostgreSQL to OneLake**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id onelake_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```


---

# Amazon Redshift Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

The credentials entries used below, and the Amazon Redshift setup they rely on, are described in [Amazon Redshift Publishing](/latest/integrations/redshift-publishing.md).

## Internal Tables[​](#internal-tables "Direct link to Internal Tables")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id s3_01 \
  --sub_path mydata \
  --publish_target redshift_classic \
  --publish_method internal \
  --publish_schema_pattern "INT_{subpath}_{date}" \
  --publish_table_pattern "{schema}_{table}"
```

## External Tables (Spectrum)[​](#external-tables-spectrum "Direct link to External Tables (Spectrum)")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id s3_01 \
  --sub_path mydata \
  --publish_target redshift_classic \
  --publish_method external \
  --publish_schema_pattern "EXT_{subpath}_{date}" \
  --publish_table_pattern "{schema}_{table}"
```

## With Primary Key Constraints[​](#with-primary-key-constraints "Direct link to With Primary Key Constraints")

Add `--pk_constraints` to include primary key definitions in internal table DDL:

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_schema_name MY_SCHEMA \
  --target_storage_id s3_01 \
  --publish_method internal \
  --publish_target redshift_classic \
  --pk_constraints
```


---

# Resume Failed Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

When a sync starts, note the run ID from output:

```
2025-10-31 10:15:23 | INFO | Created export run: 20251031-2f73b4d0-8647-11ef-8089-c403a82a4577
```

Resume with:

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id <LXDB_AUTH_ID> \
    --sync_id <SYNC_ID> \
    --run_id 20251031-2f73b4d0-8647-11ef-8089-c403a82a4577 \
    --resume
```

This skips completed tables and retries failed ones.

## Query Failed Tables[​](#query-failed-tables "Direct link to Query Failed Tables")

```
SELECT
    j.source_schema,
    j.source_table,
    j.job_status,
    l.error_message,
    j.started_at,
    j.completed_at
FROM export_jobs j
LEFT JOIN export_log l
       ON l.job_id = j.job_id AND l.error_message IS NOT NULL
WHERE j.run_id = '20251031-2f73b4d0-8647-11ef-8089-c403a82a4577'
  AND j.job_status = 'failed'
ORDER BY j.source_schema, j.source_table;
```


---

# AWS S3 Exports

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

**PostgreSQL to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**PostgreSQL to S3 (OVH)**

Uses OVH S3-compatible storage with custom endpoint via AWS profile:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**credentials.json:**

```
{
  "s3_02": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-ovh-bucket/exports",
      "profile": "ovh"
    }
  }
}
```

**\~/.aws/config:**

```
[profile ovh]
endpoint_url = https://s3.gra.io.cloud.ovh.net
region = gra
```

**\~/.aws/credentials:**

```
[ovh]
aws_access_key_id = YOUR_OVH_ACCESS_KEY
aws_secret_access_key = YOUR_OVH_SECRET_KEY
```

**PostgreSQL to S3 with Sub-Path and Table Filter**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --sub_path staging/data \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MySQL to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mysql \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --sub_path mysql_export \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MariaDB to S3**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mariadb \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --sub_path mariadb_export \
    --include "nation%" \
    --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```


---

# Snowflake Publishing

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

Export to S3, then create Snowflake tables.

## Snowflake External Tables[​](#snowflake-external-tables "Direct link to Snowflake External Tables")

Data stays in S3; queryable via Snowflake.

**External Tables with Views (Default)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path snowflake_ext \
    --include "nation%" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**External Tables without Views**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path snowflake_ext \
    --include "nation%" \
    --publish_target snowflake_prod \
    --no_views

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## Snowflake Internal Tables[​](#snowflake-internal-tables "Direct link to Snowflake Internal Tables")

Data loaded into Snowflake storage.

**Basic Internal Tables**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name "tpch_1%" \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path snowflake_int \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**Internal Tables with Custom Naming**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path production \
    --publish_method internal \
    --publish_schema_pattern "INT_{subpath}_{date}" \
    --publish_table_pattern "{schema}_{table}" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**Internal Tables with Primary Key Constraints**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path production \
    --publish_method internal \
    --publish_schema_pattern "INT_{subpath}_{date}" \
    --publish_table_pattern "{schema}_{table}" \
    --publish_target snowflake_prod \
    --pk_constraints

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**SQL Server to Snowflake Internal**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --publish_method internal \
    --publish_schema_pattern "adventureworksdw" \
    --publish_table_pattern "{table}" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MySQL to Snowflake (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mysql \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path mysql_data \
    --include "nation%" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MySQL to Snowflake (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mysql \
    --source_db_name tpch \
    --source_schema_name tpch \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path mysql_data \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MariaDB to Snowflake (External)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mariadb \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path mariadb_data \
    --include "nation%" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

**MariaDB to Snowflake (Internal)**

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mariadb \
    --source_db_name tpch_mariadb \
    --source_schema_name tpch_mariadb \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_01 \
    --generate_metadata \
    --sub_path mariadb_data \
    --include "nation%" \
    --publish_method internal \
    --publish_schema_pattern "INT_{schema}" \
    --publish_target snowflake_prod

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>
```

## More Examples[​](#more-examples "Direct link to More Examples")

The credentials entries used below, and the Snowflake setup they rely on, are described in [Snowflake Publishing](/latest/integrations/snowflake-publishing.md).

### Example 1: External Tables with Custom Patterns[​](#example-1-external-tables-with-custom-patterns "Direct link to Example 1: External Tables with Custom Patterns")

```
./LakeXpress config create -a credentials.json \
       --lxdb_auth_id lxdb_mssql \
       --source_db_auth_id source_postgres \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id s3_01 \
       --generate_metadata \
       --sub_path toto \
       --publish_target snowflake_prod \
       --publish_schema_pattern "EXT_{subpath}_{date}" \
       --publish_table_pattern "{schema}_{table}" \
       --publish_method external
```

**Result:**

* Schema: `EXT_toto_20251124`
* Tables: `tpch_1_customer`, `tpch_1_orders`, etc.
* Views: `V_tpch_1_customer`, `V_tpch_1_orders`, etc.

### Example 2: Internal Tables with Primary Keys[​](#example-2-internal-tables-with-primary-keys "Direct link to Example 2: Internal Tables with Primary Keys")

```
./LakeXpress config create -a credentials.json \
       --lxdb_auth_id lxdb_mssql \
       --source_db_auth_id source_postgres \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id s3_01 \
       --generate_metadata \
       --sub_path toto \
       --publish_method internal \
       --publish_target snowflake_prod \
       --pk_constraints
```

**Result:**

* Schema: `tpch_1`
* Tables: `customer`, `orders`, etc. (with data loaded)
* PRIMARY KEY constraints applied where available

### Example 3: Single Table[​](#example-3-single-table "Direct link to Example 3: Single Table")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --source_db_auth_id source_postgres \
  --target_storage_id s3_01 \
  --source_schema_name tpch_1 \
  -i nation \
  --publish_target snowflake_prod
```

### Example 4: Full Schema[​](#example-4-full-schema "Direct link to Example 4: Full Schema")

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_mssql \
  --source_db_auth_id source_postgres \
  --target_storage_id s3_01 \
  --source_schema_name tpch_1 \
  --publish_target snowflake_prod
```


---

# Table Filtering

Commands use Linux syntax. See [Examples & Recipes](/latest/guides/command-cookbook.md) for the Windows equivalents and the arguments every `sync` needs.

## Include Specific Table Patterns[​](#include-specific-table-patterns "Direct link to Include Specific Table Patterns")

Export only fact and dimension tables:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --include "fact_%, dim_%" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Includes `fact_sales`, `fact_orders`, `dim_customer`, `dim_product`, etc.

## Exclude Temporary and Test Tables[​](#exclude-temporary-and-test-tables "Direct link to Exclude Temporary and Test Tables")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --exclude "temp_%, test_%, staging_%" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

## Combine Include and Exclude[​](#combine-include-and-exclude "Direct link to Combine Include and Exclude")

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name analytics \
    --include "report_%, dashboard_%" \
    --exclude "%_test, %_backup" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```

Includes `report_sales`, `dashboard_kpi`. Excludes `report_sales_test`, `dashboard_kpi_backup`.

## Filter Schemas and Tables, then Publish with Custom Names[​](#filter-schemas-and-tables-then-publish-with-custom-names "Direct link to Filter Schemas and Tables, then Publish with Custom Names")

Export the `fact_` and `dim_` tables of every `sales` and `marketing` schema, skip temporary tables, and publish them to Snowflake under one schema:

```
./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name analytics \
  --source_schema_name "sales%, marketing%" \
  --include "fact_%, dim_%" \
  --exclude "temp%, test%" \
  --target_storage_id s3_01 \
  --sub_path data-lake/prod \
  --publish_target snowflake_prod \
  --publish_schema_pattern "ANALYTICS_{subpath}" \
  --publish_table_pattern "{schema}_{table}" \
  --n_jobs 8 \
  --fastbcp_p 4 \
  --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>
```


---

# Installation

## Instructions[​](#instructions "Direct link to Instructions")

* Windows
* Linux

To start using LakeXpress, first **[request a trial](https://arpe.io/get-trial?product=LakeXpress)**: you receive the download link and a free 30-day license file by email. Customers download the same package.

Once downloaded, unzip LakeXpress to a directory of your choice.

```
Expand-Archive -Path .\LakeXpress-Full-win-x64.zip -DestinationPath .\LakeXpress
```

Save the license file you received by email (`TRIAL-LakeXpress-….lic`, or `SUBSCRIPTION-LakeXpress-….lic` for customers) as `%APPDATA%\LakeXpress\license.lic`:

```
New-Item -ItemType Directory -Force "$env:APPDATA\LakeXpress" | Out-Null
Copy-Item .\*-LakeXpress-*.lic "$env:APPDATA\LakeXpress\license.lic"
```

LakeXpress does not run without it. For other ways to provide it, see [License Information](#license-information).

You can verify the installation by checking the version:

```
.\LakeXpress.exe --version
```

The output shows the version and the date the binary was built, for example `LakeXpress 0.6.3 (build 2026-11-10T14:30:06Z)`.

To start using LakeXpress, first **[request a trial](https://arpe.io/get-trial?product=LakeXpress)**: you receive the download link and a free 30-day license file by email. Customers download the same package.

Once downloaded, unzip LakeXpress to a directory of your choice.

```
unzip LakeXpress-Full-linux-x64.zip -d LakeXpress
```

On **Linux**, you need to give execution rights to the LakeXpress binary:

```
chmod +x LakeXpress
```

<br />

Save the license file you received by email (`TRIAL-LakeXpress-….lic`, or `SUBSCRIPTION-LakeXpress-….lic` for customers) as `~/.lakexpress/license.lic`:

```
mkdir -p ~/.lakexpress
cp *-LakeXpress-*.lic ~/.lakexpress/license.lic
```

LakeXpress does not run without it. For other ways to provide it, see [License Information](#license-information).

You can verify the installation by checking the version:

```
./LakeXpress --version
```

The output shows the version and the date the binary was built, for example `LakeXpress 0.6.3 (build 2026-11-10T14:30:06Z)`.

<br />

**Next step:** LakeXpress needs a database of its own to store its configurations and run history. See [Set up the LakeXpress DB](/latest/lakexpress-db/.md).

## What's Included[​](#whats-included "Direct link to What's Included")

There is a single package per platform, for the trial and for subscriptions (the license file makes the difference). It has the following layout:

* Windows
* Linux

Package layout

* `LakeXpress/`

  * `LakeXpress.exe`Main executable

  * `README.txt`Quick reference

  * `LakeXpress-sbom.*`Software Bill of Materials (JSON, XML) and checksums

  * `engine/`

    * `FastBCP.exe`auto-discoveredBundled FastBCP engine

Package layout

* `LakeXpress/`

  * `LakeXpress`Main executable

  * `README.txt`Quick reference

  * `LakeXpress-sbom.*`Software Bill of Materials (JSON, XML) and checksums

  * `engine/`

    * `FastBCP`auto-discoveredBundled FastBCP engine

Bundled FastBCP Engine

The package includes the FastBCP engine in the `engine/` directory, found automatically: no need to specify `--fastbcp_dir_path`. It is the FastBCP version this LakeXpress release was tested with. Your LakeXpress license also licenses it: no separate `FastBCP.lic` is needed.

To use another FastBCP installation, pass its folder with `--fastbcp_dir_path`.

### FastBCP version[​](#fastbcp-version "Direct link to FastBCP version")

Before each export (`sync`, `sync[export]`, `run`), LakeXpress checks the FastBCP engine:

FastBCP not configured, or no executable in the configured directory

Exit code 30`FASTBCP_NOT_FOUND`the export stops

Version older than **1.4.0**

Exit code 34`FASTBCP_VERSION_UNSUPPORTED`the export stops

Version **1.4.0**, the one bundled and tested with this LakeXpress release

OKthe export runs

Version newer than **1.4.0**, another 1.4.x patch included

Warningthe export continues

Version cannot be determined

Warningthe export continues

LakeXpress requires FastBCP 1.4.0 or later. Older versions export Timepartition tables incorrectly: they skip rows whose date is NULL, and with `week` granularity they export some rows twice.

`sync[publish]` does not run FastBCP and skips this check. See [Exit Codes](/latest/reference/exit-codes.md).

## License Information[​](#license-information "Direct link to License Information")

LakeXpress needs a license file to run, including during the free trial: the [trial request](https://arpe.io/get-trial?product=LakeXpress) emails you a 30-day trial license.

### Where LakeXpress looks for the license[​](#where-lakexpress-looks-for-the-license "Direct link to Where LakeXpress looks for the license")

LakeXpress takes the first license it finds, in this order:

1. 1`LAKEXPRESS_LICENSE` environment variableset this for CI/CD or containerized deployments
2. 2`--license TEXT`CLI argument: pass the license content directly
3. 3`--license_file PATH`CLI argument: point to a license file
4. 4Default file locations
   * `~/.lakexpress/license.lic` (Linux)
   * `%APPDATA%/LakeXpress/license.lic` (Windows)
   * `./license.lic` (current directory)

See the [CLI Reference](/latest/cli/.md#license-resolution) for details.

### FastBCP and ADBC licenses[​](#fastbcp-and-adbc-licenses "Direct link to FastBCP and ADBC licenses")

The license file also carries the FastBCP and ADBC driver licenses: keep the whole file as delivered. LakeXpress passes them to FastBCP for each export through a private temporary file that it removes at the end of the run. You do not need a separate `FastBCP.lic`.

### Missing or incomplete license[​](#missing-or-incomplete-license "Direct link to Missing or incomplete license")

No valid license found

Exit code 13`LICENSE_ERROR`LakeXpress stops

A source extracted through ADBC (the default for SQL Server, PostgreSQL and Teradata; Oracle with `use_adbc: true`), a license without the ADBC driver block, and neither `ARPEIO_ADBC_LICENCE` nor `ARPEIO_ADBC_LICENCE_FILE` set

Exit code 13the export stops

FastBCP rejects its licence during the export (missing or expired)

Exit code 13the run stops at the first table and starts no other

License issued before October 26, 2026, with no FastBCP block

WarningFastBCP uses its own FastBCP.lic, as before; request an updated license to drop it

`--help` and `--version`

OKno license needed


---

# BigQuery Publishing

LakeXpress can **automatically create BigQuery tables** from Parquet files stored in GCS.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Google Cloud Permissions[​](#1-google-cloud-permissions "Direct link to 1. Google Cloud Permissions")

Required IAM roles:

* **BigQuery Data Editor** (`roles/bigquery.dataEditor`) - Create datasets and tables
* **BigQuery Job User** (`roles/bigquery.jobUser`) - Run load jobs (native tables)
* **Storage Object Viewer** (`roles/storage.objectViewer`) - Read GCS files (external tables)

Or a custom role with these permissions:

```
bigquery.datasets.create
bigquery.datasets.get
bigquery.tables.create
bigquery.tables.get
bigquery.tables.delete
bigquery.tables.updateData
bigquery.jobs.create
storage.objects.get
storage.objects.list
```

### 2. GCS Storage Setup[​](#2-gcs-storage-setup "Direct link to 2. GCS Storage Setup")

Your `credentials.json` needs both GCS and BigQuery entries:

```
{
    "gcs_01": {
        "ds_type": "gcs",
        "auth_mode": "profile",
        "info": {
            "directory": "gs://my-datalake-bucket/lakexpress/",
            "profile": "/path/to/service-account.json"
        }
    },
    "bigquery_prod": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-gcp-project",
            "location": "US",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

The same service account JSON file can be used for both GCS and BigQuery.

## Authentication[​](#authentication "Direct link to Authentication")

LakeXpress uses service account authentication for BigQuery.

```
{
    "bigquery_prod": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-gcp-project",
            "location": "US",
            "profile": "/path/to/service-account.json"
        }
    }
}
```

| Field        | Required | Description                                   |
| ------------ | -------- | --------------------------------------------- |
| `ds_type`    | Yes      | Must be `"bigquery"`                          |
| `auth_mode`  | Yes      | Must be `"profile"`                           |
| `project_id` | Yes      | GCP project ID where datasets will be created |
| `location`   | No       | BigQuery dataset location (default: `"US"`)   |
| `profile`    | Yes      | Path to service account JSON key file         |

Common locations: `US`, `EU`, `us-central1`, `europe-west1`.

### Creating a Service Account[​](#creating-a-service-account "Direct link to Creating a Service Account")

1. Go to Google Cloud Console > IAM & Admin > Service Accounts
2. Create a service account with BigQuery and Storage permissions
3. Create and download a JSON key
4. Reference the key file path in your credentials

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                                                                                                                                    | Default                                                                                             |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `--publish_target ID`              | Credential ID for BigQuery target (required)                                                                                                                   | -                                                                                                   |
| `--publish_schema_pattern PATTERN` | BigQuery dataset naming pattern. The resolved name is lowercased, and characters other than letters, digits and `_` become `_`                                 | `{schema}` for `internal`, `EXT_{schema}` for the default `external` method (unless set explicitly) |
| `--publish_database_name NAME`     | Dataset name; replaces `--publish_schema_pattern` when set                                                                                                     | -                                                                                                   |
| `--publish_table_pattern PATTERN`  | Table naming pattern                                                                                                                                           | `{table}`                                                                                           |
| `--publish_method TYPE`            | Table type: `external` or `internal`                                                                                                                           | `external`                                                                                          |
| `--pk_constraints`                 | Add `PRIMARY KEY (...) NOT ENFORCED` to the native tables LakeXpress creates (`--publish_method internal`). An existing table kept for `append` is not changed | off                                                                                                 |
| `--n_jobs N`                       | Parallel workers for table creation                                                                                                                            | `1`                                                                                                 |

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables (Default)[​](#external-tables-default "Direct link to External Tables (Default)")

Queries data directly from GCS Parquet files without loading into BigQuery storage.

Options for `config create`:

```
--publish_target bigquery_prod
--publish_method external
```

* Fast publishing, no data copying
* No BigQuery storage costs
* Always reflects latest GCS data
* Best for large datasets with infrequent queries
* Query speed depends on GCS read performance
* No clustering or partitioning support

### Native Tables[​](#native-tables "Direct link to Native Tables")

Loads data from GCS into BigQuery's columnar storage.

Options for `config create`:

```
--publish_target bigquery_prod
--publish_method internal
```

* Faster queries via optimized columnar storage
* Full BigQuery features (clustering, partitioning, materialized views)
* Best for frequently queried data
* Slower publishing due to data loading
* Incurs BigQuery storage costs
* Data is a point-in-time snapshot

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Dataset and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                           | Example Output    |
| ------------- | ------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                    | `tpch_1`          |
| `{table}`     | Source table name                     | `customer`        |
| `{date}`      | Current date (YYYYMMDD)               | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  | `20251210_143022` |
| `{uuid}`      | UUID4 identifier (consistent per run) | `a1b2c3d4`        |
| `{subpath}`   | CLI `--sub_path` parameter            | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Prefixed Datasets

```
--publish_schema_pattern "lx_{schema}"
--publish_table_pattern "{table}"
--publish_target bigquery_prod

# Results:
# Dataset: lx_tpch_1
# Tables: customer, orders, lineitem
```

Date-Partitioned Datasets

```
--publish_schema_pattern "{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target bigquery_prod

# Results:
# Dataset: tpch_1_20251210
# Tables: customer, orders, lineitem
```

Consolidated Multi-Schema

```
--source_schema_name schema1,schema2
--publish_schema_pattern "consolidated"
--publish_table_pattern "{schema}_{table}"
--publish_target bigquery_prod

# Results:
# Dataset: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for BigQuery are in [Examples & Recipes: BigQuery](/latest/guides/recipes/bigquery.md).

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. `upsert` and `delete_insert` need a primary key and a native table (`--publish_method internal`). See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/bigquery.svg)BigQuery Reference](/latest/integrations/bigquery-reference.md)Data type mappings, CLI arguments, querying, troubleshooting
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# BigQuery Reference

Reference material for [BigQuery Publishing](/latest/integrations/bigquery-publishing.md) -- data type mappings, CLI arguments, querying, and troubleshooting.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to BigQuery-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to BigQuery[​](#postgresql-to-bigquery "Direct link to PostgreSQL to BigQuery")

| PostgreSQL Type                                     | BigQuery Type |
| --------------------------------------------------- | ------------- |
| `SMALLINT`, `INTEGER`, `INT4`, `BIGINT`, `INT8`     | `INT64`       |
| `NUMERIC(p,s)`                                      | `NUMERIC`     |
| `REAL`, `FLOAT4`, `DOUBLE PRECISION`, `FLOAT8`      | `FLOAT64`     |
| `VARCHAR(n)`, `TEXT`, `CHAR(n)`, `UUID`, `array []` | `STRING`      |
| `BOOLEAN`                                           | `BOOL`        |
| `DATE`                                              | `DATE`        |
| `TIME`                                              | `TIME`        |
| `TIMESTAMP`, `TIMESTAMPTZ`                          | `TIMESTAMP`   |
| `BYTEA`                                             | `BYTES`       |
| `JSON`, `JSONB`                                     | `JSON`        |

### SQL Server to BigQuery[​](#sql-server-to-bigquery "Direct link to SQL Server to BigQuery")

| SQL Server Type                                                                         | BigQuery Type |
| --------------------------------------------------------------------------------------- | ------------- |
| `TINYINT`, `SMALLINT`, `INT`, `BIGINT`                                                  | `INT64`       |
| `DECIMAL(p,s)`, `MONEY`, `SMALLMONEY`                                                   | `NUMERIC`     |
| `FLOAT`, `REAL`                                                                         | `FLOAT64`     |
| `BIT`                                                                                   | `BOOL`        |
| `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `TEXT`, `NTEXT`, `UNIQUEIDENTIFIER` | `STRING`      |
| `DATE`                                                                                  | `DATE`        |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME`                                                | `DATETIME`    |
| `DATETIMEOFFSET`                                                                        | `TIMESTAMP`   |
| `VARBINARY`                                                                             | `BYTES`       |

### Oracle to BigQuery[​](#oracle-to-bigquery "Direct link to Oracle to BigQuery")

| Oracle Type                                               | BigQuery Type |
| --------------------------------------------------------- | ------------- |
| `NUMBER`, `NUMBER(p,s)`                                   | `NUMERIC`     |
| `FLOAT`                                                   | `FLOAT64`     |
| `VARCHAR2(n)`, `NVARCHAR2(n)`, `CHAR(n)`, `CLOB`, `NCLOB` | `STRING`      |
| `DATE`                                                    | `DATE`        |
| `TIMESTAMP`                                               | `TIMESTAMP`   |
| `RAW`, `BLOB`                                             | `BYTES`       |

### Teradata to BigQuery[​](#teradata-to-bigquery "Direct link to Teradata to BigQuery")

| Teradata Type                                         | BigQuery Type |
| ----------------------------------------------------- | ------------- |
| `BYTEINT`, `VARCHAR(n)`, `CHAR(n)`, `CLOB`, `VARBYTE` | `STRING`      |
| `SMALLINT`, `INTEGER`, `BIGINT`                       | `INT64`       |
| `DECIMAL(p,s)`, `NUMERIC(p,s)`, `NUMBER`              | `NUMERIC`     |
| `FLOAT`, `REAL`                                       | `FLOAT64`     |
| `DATE`                                                | `DATE`        |
| `TIME`                                                | `TIME`        |
| `TIMESTAMP`                                           | `TIMESTAMP`   |
| `BYTE`, `BLOB`                                        | `BYTES`       |

### MySQL to BigQuery[​](#mysql-to-bigquery "Direct link to MySQL to BigQuery")

| MySQL Type                                                                                                                                      | BigQuery Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `TINYINT`, `SMALLINT`, `INT`, `BIGINT`                                                                                                          | `INT64`       |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `STRING`      |
| `DECIMAL(p,s)`                                                                                                                                  | `NUMERIC`     |
| `FLOAT`, `DOUBLE`                                                                                                                               | `FLOAT64`     |
| `BIT`                                                                                                                                           | `BOOL`        |
| `DATE`                                                                                                                                          | `DATE`        |
| `TIME`                                                                                                                                          | `TIME`        |
| `DATETIME`                                                                                                                                      | `DATETIME`    |
| `TIMESTAMP`                                                                                                                                     | `TIMESTAMP`   |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                   | `BYTES`       |
| `JSON`                                                                                                                                          | `JSON`        |

### MariaDB to BigQuery[​](#mariadb-to-bigquery "Direct link to MariaDB to BigQuery")

| MariaDB Type                                                                                                                                    | BigQuery Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `TINYINT`, `SMALLINT`, `INT`, `BIGINT`                                                                                                          | `INT64`       |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `STRING`      |
| `DECIMAL(p,s)`                                                                                                                                  | `NUMERIC`     |
| `FLOAT`, `DOUBLE`                                                                                                                               | `FLOAT64`     |
| `BIT`                                                                                                                                           | `BOOL`        |
| `DATE`                                                                                                                                          | `DATE`        |
| `TIME`                                                                                                                                          | `TIME`        |
| `DATETIME`                                                                                                                                      | `DATETIME`    |
| `TIMESTAMP`                                                                                                                                     | `TIMESTAMP`   |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                   | `BYTES`       |
| `JSON`                                                                                                                                          | `JSON`        |

### SAP HANA to BigQuery[​](#sap-hana-to-bigquery "Direct link to SAP HANA to BigQuery")

| SAP HANA Type                                                                                             | BigQuery Type |
| --------------------------------------------------------------------------------------------------------- | ------------- |
| `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`                                                                | `INT64`       |
| `DECIMAL(p,s)`                                                                                            | `NUMERIC`     |
| `SMALLDECIMAL`, `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `CLOB`, `NCLOB`, `TEXT`, `SECONDDATE` | `STRING`      |
| `REAL`, `DOUBLE`                                                                                          | `FLOAT64`     |
| `BOOLEAN`                                                                                                 | `BOOL`        |
| `DATE`                                                                                                    | `DATE`        |
| `TIME`                                                                                                    | `TIME`        |
| `TIMESTAMP`                                                                                               | `TIMESTAMP`   |
| `VARBINARY`, `BLOB`                                                                                       | `BYTES`       |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### BigQuery Publishing Arguments[​](#bigquery-publishing-arguments "Direct link to BigQuery Publishing Arguments")

| Option                             | Type    | Description                                                                                                                                            |
| ---------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--publish_target ID`              | String  | Credential ID for BigQuery publishing (required)                                                                                                       |
| `--publish_schema_pattern PATTERN` | String  | Dynamic dataset naming pattern (default: `{schema}` for `internal`, `EXT_{schema}` for the default `external` method; the resolved name is lowercased) |
| `--publish_database_name NAME`     | String  | Dataset name; replaces `--publish_schema_pattern` when set                                                                                             |
| `--publish_table_pattern PATTERN`  | String  | Dynamic table naming pattern (default: `{table}`)                                                                                                      |
| `--publish_method TYPE`            | String  | Table type: `external` (default) or `internal`                                                                                                         |
| `--pk_constraints`                 | Flag    | Add `PRIMARY KEY (...) NOT ENFORCED` to the native tables LakeXpress creates (`--publish_method internal`)                                             |
| `--n_jobs N`                       | Integer | Parallel workers for table creation (default: 1)                                                                                                       |

### Querying BigQuery Tables[​](#querying-bigquery-tables "Direct link to Querying BigQuery Tables")

**BigQuery Console:**

```
SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10;
```

**bq CLI:**

```
bq query --use_legacy_sql=false \
  'SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10'
```

**Python (google-cloud-bigquery):**

```
from google.cloud import bigquery

client = bigquery.Client()
query = "SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10"
df = client.query(query).to_dataframe()
print(df)
```

**pandas-gbq:**

```
import pandas_gbq

query = "SELECT * FROM `my-project.lx_tpch_1.customer` LIMIT 10"
df = pandas_gbq.read_gbq(query, project_id="my-project")
print(df)
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

**"Permission denied" errors:**

* Verify the service account has BigQuery Data Editor role
* Check GCS bucket access from the service account
* Ensure project ID in credentials matches the target project

**"Dataset not found" errors:**

* Check that `location` matches your GCS bucket region
* Cross-region access between GCS and BigQuery may cause issues

**"Invalid table" errors for external tables:**

* Verify GCS path contains valid Parquet files
* Check schema mapping for your data types

### Verifying Setup[​](#verifying-setup "Direct link to Verifying Setup")

Test BigQuery connectivity by creating a minimal sync configuration against a single schema, then run it and check the log output:

```
./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target bigquery_prod \
    --log_level DEBUG
```

There is no `--dry_run` flag for BigQuery publishing runs -- to limit scope for a connectivity check, point `--source_schema_name` at a schema with few tables, or select tables with `config create -i/--include` (see the [CLI Reference](/latest/cli/.md)).

Validate credentials:

```
from google.cloud import bigquery
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file(
    '/path/to/service-account.json'
)
client = bigquery.Client(credentials=credentials, project='my-project')
print(list(client.list_datasets()))
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/bigquery.svg)BigQuery Publishing](/latest/integrations/bigquery-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options


---

# Databricks Publishing

LakeXpress creates **Databricks Unity Catalog tables** from exported Parquet files. Supports external tables (data stays in S3) and managed Delta tables (data loaded into Databricks).

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Databricks Setup[​](#1-databricks-setup "Direct link to 1. Databricks Setup")

**Required Resources:**

* Databricks Workspace with Unity Catalog enabled
* SQL Warehouse (for DDL execution)
* Catalog with CREATE SCHEMA and CREATE TABLE permissions
* Storage credential for S3 access (external tables only)

**Required Permissions:**

* `CREATE SCHEMA` on the target catalog
* `CREATE TABLE` on the target schema
* `USE CATALOG` on the target catalog
* For external tables: access to the external location

### 2. Storage Credential Setup[​](#2-storage-credential-setup "Direct link to 2. Storage Credential Setup")

For external tables, configure a storage credential for S3 access:

```
-- In Databricks SQL
CREATE STORAGE CREDENTIAL my_s3_credential
WITH (
    AWS_IAM_ROLE = 'arn:aws:iam::123456789012:role/DatabricksS3AccessRole'
);

CREATE EXTERNAL LOCATION my_s3_location
URL 's3://my-datalake-bucket/lakexpress/'
WITH (STORAGE CREDENTIAL my_s3_credential);
```

### 3. Credentials Configuration[​](#3-credentials-configuration "Direct link to 3. Credentials Configuration")

Both S3 and Databricks credentials are needed in `credentials.json`:

```
{
    "s3_01": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://your-bucket-name/path/to/exports",
            "profile": "your-aws-profile"
        }
    },
    "databricks_prod": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}
```

## Authentication[​](#authentication "Direct link to Authentication")

Databricks uses Personal Access Tokens (PAT):

```
{
    "databricks_prod": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}
```

**Configuration Fields:**

| Field          | Description                   | Required |
| -------------- | ----------------------------- | -------- |
| `host`         | Databricks workspace hostname | Yes      |
| `http_path`    | SQL Warehouse HTTP path       | Yes      |
| `access_token` | Personal Access Token (PAT)   | Yes      |
| `catalog`      | Target Unity Catalog          | Yes      |
| `schema`       | Default schema                | No       |

**To get the HTTP path:**

1. Go to Databricks SQL > SQL Warehouses
2. Select your warehouse > Connection details
3. Copy the HTTP path

**To create a PAT:**

1. Go to User Settings > Developer > Access tokens
2. Generate a new token

**Authentication mode:** only `token` is supported. `auth_mode: "oauth"` is rejected with an error: there is no OAuth flow. For a service principal, put its token in `access_token`.

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables[​](#external-tables "Direct link to External Tables")

Reference Parquet data in S3. Databricks stores only metadata; queries read directly from S3.

* No data loading time or Databricks storage costs
* Data stays in place
* Available right after export

Options for `config create`:

```
--publish_target databricks_prod
--publish_method external  # Default
```

### Managed Tables (Delta)[​](#managed-tables-delta "Direct link to Managed Tables (Delta)")

Delta tables loaded into Databricks managed storage via COPY INTO. Enables ACID transactions, time travel, and Z-ordering.

* Faster queries with optimized storage and caching
* Full Delta Lake features
* Better for frequently accessed data

Options for `config create`:

```
--publish_target databricks_prod
--publish_method internal  # Databricks maps "internal" to managed Delta tables
```

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                                 | Description                                                                   | Default                                              |
| -------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------- |
| `--publish_target ID`                  | Credential ID for Databricks target (required)                                | -                                                    |
| `--publish_schema_pattern PATTERN`     | Schema naming pattern                                                         | `EXT_{schema}` for external, `{schema}` for internal |
| `--publish_table_pattern PATTERN`      | Table naming pattern                                                          | `{table}`                                            |
| `--publish_method {external,internal}` | `external` (Unity Catalog external table) or `internal` (managed Delta table) | `external`                                           |
| `--pk_constraints`                     | Add PRIMARY KEY constraints (managed/Delta tables only)                       | Disabled                                             |
| `--n_jobs N`                           | Parallel workers for table creation                                           | `1`                                                  |

There is no `--databricks_catalog` flag -- the target catalog always comes from the `catalog` field of the Databricks credential.

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                          | Example Output    |
| ------------- | ------------------------------------ | ----------------- |
| `{schema}`    | Source schema name                   | `tpch_1`          |
| `{table}`     | Source table name                    | `customer`        |
| `{date}`      | Current date (YYYYMMDD)              | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS) | `20251210_143022` |
| `{uuid}`      | UUID4, consistent per run            | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` value               | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Prefixed Schemas

```
--publish_schema_pattern "bronze_{schema}"
--publish_table_pattern "{table}"
--publish_target databricks_prod

# Result:
# lakexpress_catalog.bronze_tpch_1.customer
# lakexpress_catalog.bronze_tpch_1.orders
```

Date-Based Schemas

```
--publish_schema_pattern "{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target databricks_prod

# Result:
# lakexpress_catalog.tpch_1_20251210.customer
```

Medallion Architecture

```
# Bronze layer (raw data, external tables)
--publish_schema_pattern "bronze_{schema}"
--publish_method external
--publish_target databricks_prod

# Silver layer (curated data, managed Delta tables)
--publish_schema_pattern "silver_{schema}"
--publish_method internal
--publish_target databricks_prod
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for Databricks are in [Examples & Recipes: Databricks](/latest/guides/recipes/databricks.md).

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. `upsert` and `delete_insert` need a primary key and a managed table (`--publish_method internal`). See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/databricks.svg)Databricks Reference](/latest/integrations/databricks-reference.md)Data type mappings, CLI arguments, querying, Delta features
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# Databricks Reference

Reference material for [Databricks Publishing](/latest/integrations/databricks-publishing.md) -- data type mappings, CLI arguments, and querying.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to Databricks-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to Databricks[​](#postgresql-to-databricks "Direct link to PostgreSQL to Databricks")

| PostgreSQL Type                                     | Databricks Type |
| --------------------------------------------------- | --------------- |
| `SMALLINT`                                          | `SMALLINT`      |
| `INTEGER`, `INT4`                                   | `INT`           |
| `BIGINT`, `INT8`                                    | `BIGINT`        |
| `NUMERIC(p,s)`                                      | `DECIMAL(p,s)`  |
| `REAL`, `FLOAT4`                                    | `FLOAT`         |
| `DOUBLE PRECISION`, `FLOAT8`                        | `DOUBLE`        |
| `VARCHAR(n)`                                        | `VARCHAR(n)`    |
| `TEXT`, `TIME`, `UUID`, `JSON`, `JSONB`, `array []` | `STRING`        |
| `CHAR(n)`                                           | `CHAR(n)`       |
| `BOOLEAN`                                           | `BOOLEAN`       |
| `DATE`                                              | `DATE`          |
| `TIMESTAMP`                                         | `TIMESTAMP_NTZ` |
| `TIMESTAMPTZ`                                       | `TIMESTAMP`     |
| `BYTEA`                                             | `BINARY`        |

### SQL Server to Databricks[​](#sql-server-to-databricks "Direct link to SQL Server to Databricks")

| SQL Server Type                          | Databricks Type |
| ---------------------------------------- | --------------- |
| `TINYINT`                                | `TINYINT`       |
| `SMALLINT`                               | `SMALLINT`      |
| `INT`                                    | `INT`           |
| `BIGINT`                                 | `BIGINT`        |
| `DECIMAL(p,s)`                           | `DECIMAL(p,s)`  |
| `MONEY`                                  | `DECIMAL(19,4)` |
| `SMALLMONEY`                             | `DECIMAL(10,4)` |
| `FLOAT`                                  | `DOUBLE`        |
| `REAL`                                   | `FLOAT`         |
| `BIT`                                    | `BOOLEAN`       |
| `VARCHAR(n)`, `NVARCHAR(n)`              | `VARCHAR(n)`    |
| `CHAR(n)`, `NCHAR(n)`                    | `CHAR(n)`       |
| `TEXT`, `NTEXT`, `UNIQUEIDENTIFIER`      | `STRING`        |
| `DATE`                                   | `DATE`          |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME` | `TIMESTAMP_NTZ` |
| `DATETIMEOFFSET`                         | `TIMESTAMP`     |
| `VARBINARY`                              | `BINARY`        |

### Oracle to Databricks[​](#oracle-to-databricks "Direct link to Oracle to Databricks")

| Oracle Type                   | Databricks Type  |
| ----------------------------- | ---------------- |
| `NUMBER`                      | `DECIMAL(38,18)` |
| `NUMBER(p,s)`                 | `DECIMAL(p,s)`   |
| `FLOAT`                       | `DOUBLE`         |
| `VARCHAR2(n)`, `NVARCHAR2(n)` | `VARCHAR(n)`     |
| `CHAR(n)`                     | `CHAR(n)`        |
| `CLOB`, `NCLOB`               | `STRING`         |
| `DATE`                        | `DATE`           |
| `TIMESTAMP`                   | `TIMESTAMP_NTZ`  |
| `RAW`, `BLOB`                 | `BINARY`         |

### Teradata to Databricks[​](#teradata-to-databricks "Direct link to Teradata to Databricks")

| Teradata Type                        | Databricks Type  |
| ------------------------------------ | ---------------- |
| `BYTEINT`, `CLOB`, `TIME`, `VARBYTE` | `STRING`         |
| `SMALLINT`                           | `SMALLINT`       |
| `INTEGER`                            | `INT`            |
| `BIGINT`                             | `BIGINT`         |
| `DECIMAL(p,s)`, `NUMERIC(p,s)`       | `DECIMAL(p,s)`   |
| `NUMBER`                             | `DECIMAL(38,18)` |
| `FLOAT`, `REAL`                      | `DOUBLE`         |
| `VARCHAR(n)`                         | `VARCHAR(n)`     |
| `CHAR(n)`                            | `CHAR(n)`        |
| `DATE`                               | `DATE`           |
| `TIMESTAMP`                          | `TIMESTAMP_NTZ`  |
| `BYTE`, `BLOB`                       | `BINARY`         |

### MySQL to Databricks[​](#mysql-to-databricks "Direct link to MySQL to Databricks")

| MySQL Type                                                                  | Databricks Type |
| --------------------------------------------------------------------------- | --------------- |
| `TINYINT`                                                                   | `TINYINT`       |
| `SMALLINT`, `YEAR`                                                          | `SMALLINT`      |
| `MEDIUMINT`, `INT`                                                          | `INT`           |
| `BIGINT`                                                                    | `BIGINT`        |
| `DECIMAL(p,s)`                                                              | `DECIMAL(p,s)`  |
| `FLOAT`, `DOUBLE`                                                           | `DOUBLE`        |
| `BIT`                                                                       | `BOOLEAN`       |
| `VARCHAR(n)`                                                                | `VARCHAR(n)`    |
| `CHAR(n)`                                                                   | `CHAR(n)`       |
| `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TIME`, `JSON` | `STRING`        |
| `DATE`                                                                      | `DATE`          |
| `DATETIME`, `TIMESTAMP`                                                     | `TIMESTAMP_NTZ` |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`         | `BINARY`        |

### MariaDB to Databricks[​](#mariadb-to-databricks "Direct link to MariaDB to Databricks")

| MariaDB Type                                                                | Databricks Type |
| --------------------------------------------------------------------------- | --------------- |
| `TINYINT`                                                                   | `TINYINT`       |
| `SMALLINT`, `YEAR`                                                          | `SMALLINT`      |
| `MEDIUMINT`, `INT`                                                          | `INT`           |
| `BIGINT`                                                                    | `BIGINT`        |
| `DECIMAL(p,s)`                                                              | `DECIMAL(p,s)`  |
| `FLOAT`, `DOUBLE`                                                           | `DOUBLE`        |
| `BIT`                                                                       | `BOOLEAN`       |
| `VARCHAR(n)`                                                                | `VARCHAR(n)`    |
| `CHAR(n)`                                                                   | `CHAR(n)`       |
| `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TIME`, `JSON` | `STRING`        |
| `DATE`                                                                      | `DATE`          |
| `DATETIME`, `TIMESTAMP`                                                     | `TIMESTAMP_NTZ` |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`         | `BINARY`        |

### SAP HANA to Databricks[​](#sap-hana-to-databricks "Direct link to SAP HANA to Databricks")

| SAP HANA Type                                                 | Databricks Type |
| ------------------------------------------------------------- | --------------- |
| `TINYINT`                                                     | `TINYINT`       |
| `SMALLINT`                                                    | `SMALLINT`      |
| `INTEGER`                                                     | `INT`           |
| `BIGINT`                                                      | `BIGINT`        |
| `DECIMAL(p,s)`                                                | `DECIMAL(p,s)`  |
| `SMALLDECIMAL`, `CLOB`, `NCLOB`, `TEXT`, `TIME`, `SECONDDATE` | `STRING`        |
| `REAL`                                                        | `FLOAT`         |
| `DOUBLE`                                                      | `DOUBLE`        |
| `BOOLEAN`                                                     | `BOOLEAN`       |
| `VARCHAR(n)`, `NVARCHAR(n)`                                   | `VARCHAR(n)`    |
| `CHAR(n)`, `NCHAR(n)`                                         | `CHAR(n)`       |
| `DATE`                                                        | `DATE`          |
| `TIMESTAMP`                                                   | `TIMESTAMP_NTZ` |
| `VARBINARY`, `BLOB`                                           | `BINARY`        |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Databricks Publishing Arguments[​](#databricks-publishing-arguments "Direct link to Databricks Publishing Arguments")

| Option                                 | Type    | Description                                                                                         |
| -------------------------------------- | ------- | --------------------------------------------------------------------------------------------------- |
| `--publish_target ID`                  | String  | Credential ID for Databricks publishing (required)                                                  |
| `--publish_schema_pattern PATTERN`     | String  | Schema naming pattern (default: `EXT_{schema}` for external, `{schema}` for internal)               |
| `--publish_table_pattern PATTERN`      | String  | Table naming pattern (default: `{table}`)                                                           |
| `--publish_method {external,internal}` | Enum    | `external` (Unity Catalog external table) or `internal` (managed Delta table) (default: `external`) |
| `--pk_constraints`                     | Flag    | Add PRIMARY KEY constraints (managed/Delta tables only)                                             |
| `--n_jobs N`                           | Integer | Parallel workers for table creation (default: 1)                                                    |

There is no `--databricks_catalog` flag. The target catalog is always taken from the `catalog` field of the Databricks credential in `credentials.json`.

### Deprecated / Renamed Arguments[​](#deprecated--renamed-arguments "Direct link to Deprecated / Renamed Arguments")

| Old Argument              | Current Argument                        | Notes                                                                                     |
| ------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `--databricks_table_type` | `--publish_method {external\|internal}` | Renamed to the generic, cross-platform flag (`internal` maps to `managed` for Databricks) |
| `--databricks_catalog`    | *(none)*                                | Never implemented as a CLI flag; the catalog comes from credentials only                  |

### Querying Tables[​](#querying-tables "Direct link to Querying Tables")

**Databricks SQL:**

```
SELECT * FROM lakexpress_catalog.tpch_1.customer LIMIT 10;
```

**Databricks Notebooks (Python):**

```
df = spark.table("lakexpress_catalog.tpch_1.customer")
df.display()
```

**Databricks Notebooks (SQL):**

```
%sql
SELECT * FROM lakexpress_catalog.tpch_1.customer LIMIT 10
```

### Delta Table Features (Managed Tables Only)[​](#delta-table-features-managed-tables-only "Direct link to Delta Table Features (Managed Tables Only)")

**Time Travel:**

```
SELECT * FROM lakexpress_catalog.tpch_1.customer VERSION AS OF 1;
```

**Table History:**

```
DESCRIBE HISTORY lakexpress_catalog.tpch_1.customer;
```

**Optimize:**

```
OPTIMIZE lakexpress_catalog.tpch_1.customer;
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/databricks.svg)Databricks Publishing](/latest/integrations/databricks-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options


---

# DuckLake Publishing

LakeXpress registers exported Parquet files as tables in a [DuckLake](https://ducklake.select/) catalog **without copying data** -- the catalog stores metadata (schema, file locations, statistics) while the Parquet files stay in place.

## What is DuckLake?[​](#what-is-ducklake "Direct link to What is DuckLake?")

[DuckLake](https://ducklake.select/) is an open table format and DuckDB extension that stores table metadata (schema, snapshots, file listings, statistics) in a SQL catalog database, while the actual data lives as Parquet files in local or cloud storage. It provides ACID transactions, time travel, and schema evolution on top of plain Parquet -- without requiring a dedicated metadata service.

LakeXpress publishes to DuckLake by creating one table per exported source table and registering the corresponding Parquet files with the catalog, so the data can be queried with standard SQL through DuckDB.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. The `ducklake` DuckDB extension[​](#1-the-ducklake-duckdb-extension "Direct link to 1-the-ducklake-duckdb-extension")

LakeXpress runs `INSTALL ducklake` and `LOAD ducklake` automatically before attaching the catalog -- this is a core DuckDB extension (not a community extension), so no manual setup is required as long as the machine running LakeXpress can install it (or already has it cached).

### 2. A catalog database[​](#2-a-catalog-database "Direct link to 2. A catalog database")

DuckLake stores its metadata in a separate database, which LakeXpress calls the **catalog**. The catalog credential entry in `credentials.json` must use one of the supported catalog backends: `sqlite`, `duckdb`, `postgres`, or `mysql`.

### 3. An intermediate storage location for the Parquet files[​](#3-an-intermediate-storage-location-for-the-parquet-files "Direct link to 3. An intermediate storage location for the Parquet files")

The exported Parquet files (local directory or cloud storage such as S3, GCS, or Azure) that were produced by the export step -- referenced via `--target_storage_id` or `--output_dir` -- are the files registered into the DuckLake catalog.

## Catalog Configuration[​](#catalog-configuration "Direct link to Catalog Configuration")

The credential entry pointed to by `--publish_target` **is** the DuckLake catalog: its `ds_type` must be one of the four supported catalog backends. LakeXpress detects a DuckLake target automatically when `ds_type` is `sqlite`, `duckdb`, `postgres`, or `mysql` (or when the entry explicitly sets `"catalog_type": "ducklake"`).

### SQLite catalog[​](#sqlite-catalog "Direct link to SQLite catalog")

```
{
    "ducklake_catalog": {
        "ds_type": "sqlite",
        "info": {
            "filepath": "/path/to/ducklake_catalog.sqlite"
        }
    }
}
```

### DuckDB catalog[​](#duckdb-catalog "Direct link to DuckDB catalog")

```
{
    "ducklake_catalog": {
        "ds_type": "duckdb",
        "info": {
            "filepath": "/path/to/ducklake_catalog.duckdb"
        }
    }
}
```

### PostgreSQL catalog[​](#postgresql-catalog "Direct link to PostgreSQL catalog")

```
{
    "ducklake_catalog": {
        "ds_type": "postgres",
        "info": {
            "server": "my-postgres-host",
            "port": 5432,
            "database": "ducklake_meta",
            "username": "ducklake_user",
            "password": "secret",
            "schema": "my_schema"
        }
    }
}
```

### MySQL catalog[​](#mysql-catalog "Direct link to MySQL catalog")

```
{
    "ducklake_catalog": {
        "ds_type": "mysql",
        "info": {
            "server": "my-mysql-host",
            "port": 3306,
            "database": "ducklake_meta",
            "username": "ducklake_user",
            "password": "secret"
        }
    }
}
```

| Field                             | Backends            | Required | Description                                                                                                                                                                           |
| --------------------------------- | ------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ds_type`                         | all                 | Yes      | One of `sqlite`, `duckdb`, `postgres`, `mysql`                                                                                                                                        |
| `info.filepath`                   | `sqlite`, `duckdb`  | No       | Path to the catalog file. Defaults to `ducklake_catalog.sqlite` / `ducklake_catalog.duckdb` in the working directory                                                                  |
| `info.server`                     | `postgres`, `mysql` | No       | Host name. Default: `localhost`                                                                                                                                                       |
| `info.port`                       | `postgres`, `mysql` | No       | Port. Default: `5432` (postgres) / `3306` (mysql)                                                                                                                                     |
| `info.database`                   | `postgres`, `mysql` | No       | Database name. Default: `postgres` (postgres backend only)                                                                                                                            |
| `info.username` / `info.password` | `postgres`, `mysql` | Yes      | Catalog database credentials                                                                                                                                                          |
| `info.schema`                     | `postgres` only     | No       | Schema used for the catalog's own metadata tables (set via `search_path`). Ignored for `mysql` and `sqlite`; `duckdb` handles schema at query time. Not passed for the other backends |

## Usage Example[​](#usage-example "Direct link to Usage Example")

Ready-to-run commands for DuckLake are in [Examples & Recipes: DuckLake](/latest/guides/recipes/ducklake.md).

## Querying DuckLake Tables[​](#querying-ducklake-tables "Direct link to Querying DuckLake Tables")

Once published, query the tables with DuckDB using the same catalog connection string used for publishing:

```
INSTALL ducklake;
LOAD ducklake;

ATTACH 'sqlite:/path/to/ducklake_catalog.sqlite' AS lakexpress_lake (TYPE ducklake);

SELECT * FROM lakexpress_lake.my_schema.my_table LIMIT 10;
```

For a PostgreSQL catalog, the attach string instead looks like:

```
ATTACH 'postgres:dbname=ducklake_meta host=my-postgres-host port=5432 user=ducklake_user password=secret'
    AS lakexpress_lake (TYPE ducklake);
```

## Notes and Limitations[​](#notes-and-limitations "Direct link to Notes and Limitations")

* **No data copy** (default `append` strategy): DuckLake tables are registered against the existing exported Parquet files (`CALL ducklake_add_data_files(...)`) -- the data is not moved or duplicated. Tables configured with the `upsert` or `delete_insert` strategy are the exception: their rows are copied into a DuckLake-managed table (delete by primary key, then insert).
* **Default schema pattern**: `EXT_{schema}`, or `{schema}` with `--publish_method internal`, unless `--publish_schema_pattern` is set. `--publish_method` has no other effect on DuckLake.
* **Supported catalog backends**: `sqlite`, `duckdb`, `postgres`, and `mysql` only. Any other `ds_type` on the `--publish_target` credential is rejected.
* **Default schema name**: if the catalog credential does not set `info.schema`, LakeXpress logs and tracks the catalog schema as `main` (used for run tracking metadata; it does not affect the target schema/table naming, which is controlled by `--publish_schema_pattern` / `--publish_table_pattern`).
* **Table replacement**: if a target table already exists, LakeXpress drops and recreates it before re-registering files (matching the behavior of the other publishers).
* **Column types**: when the exported Parquet file is reachable, LakeXpress derives column types directly from the file (`DESCRIBE SELECT * FROM read_parquet(...)`); otherwise it falls back to mapping the source database types to DuckDB types from the export metadata.
* **Sequential publishing**: table creation currently runs sequentially regardless of `--n_jobs`.

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. A table using `upsert` or `delete_insert` needs a primary key, and its rows are copied into a DuckLake-managed table. DuckLake has no `MERGE`: both strategies delete the matching rows, then insert. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [Storage Configuration Overview](/latest/configuration/storage/storage-overview.md)Configuring S3, GCS, Azure, and local storage
* [CLI Reference](/latest/cli/.md)All command-line options
* [![](/img/logos/motherduck.svg)MotherDuck Publishing](/latest/integrations/motherduck-publishing.md)Sibling DuckDB-family publishing target
* [![](/img/logos/motherduck.svg)MotherDuck Reference](/latest/integrations/motherduck-reference.md)Data type mappings and querying patterns


---

# Microsoft Fabric Publishing

LakeXpress creates Microsoft Fabric Lakehouse tables from exported Parquet files. Query data in Fabric using SQL, Spark, or Power BI via managed Delta tables or SQL analytics views.

**Note:** LakeXpress supports two distinct Fabric publish targets, selected by the credential's `fabric_target` field: `"lakehouse"` (Delta tables / OneLake, covered by this page) and the Fabric **Warehouse** (T-SQL) target, which is used whenever `fabric_target` is omitted or set to anything else. Make sure your `credentials.json` sets `"fabric_target": "lakehouse"` explicitly if you want the behavior documented here.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Microsoft Fabric Setup[​](#1-microsoft-fabric-setup "Direct link to 1. Microsoft Fabric Setup")

**Required Resources:**

* Microsoft Fabric workspace with a Lakehouse
* Service Principal with appropriate permissions
* OneLake storage access

**Required Permissions:**

* **Workspace**: Member or Contributor role
* **Lakehouse**: Read and Write access
* **OneLake**: Storage Blob Data Contributor

### 2. Azure AD App Registration[​](#2-azure-ad-app-registration "Direct link to 2. Azure AD App Registration")

Create a Service Principal:

1. **Azure Portal** > **Azure Active Directory** > **App registrations**
2. Click **New registration**
3. Name the app (e.g., "LakeXpress-Fabric")
4. Select **Accounts in this organizational directory only**
5. Click **Register**
6. Note the **Application (client) ID** and **Directory (tenant) ID**
7. Go to **Certificates & secrets** > **New client secret**
8. Create a secret and save the **Value** (shown only once)

### 3. Grant Fabric Permissions[​](#3-grant-fabric-permissions "Direct link to 3. Grant Fabric Permissions")

Add the Service Principal to your workspace:

1. Open your **Fabric workspace**
2. Click **Manage access**
3. Click **Add people or groups**
4. Search for your Service Principal
5. Assign **Member** or **Contributor** role
6. Click **Add**

### 4. Find Fabric Configuration Values[​](#4-find-fabric-configuration-values "Direct link to 4. Find Fabric Configuration Values")

**Workspace ID and Lakehouse ID:**

1. Open your Lakehouse in the Fabric portal
2. Extract from the URL:
   <!-- -->
   ```
   https://app.fabric.microsoft.com/groups/{workspace_id}/lakehouses/{lakehouse_id}
   ```
3. Copy `workspace_id` (GUID after `/groups/`)
4. Copy `lakehouse_id` (GUID after `/lakehouses/`)

**Lakehouse Name:**

The display name shown in the Fabric portal.

**SQL Analytics Endpoint:**

1. In your Lakehouse, click **SQL analytics endpoint** in the bottom pane
2. Click **Copy SQL connection string**
3. Extract the hostname (e.g., `abc123xyz.datawarehouse.fabric.microsoft.com`)

### 5. Credentials Configuration[​](#5-credentials-configuration "Direct link to 5. Credentials Configuration")

Both OneLake storage and Fabric publishing credentials go in `credentials.json`:

```
{
    "onelake_01": {
        "ds_type": "onelake",
        "auth_mode": "service_principal",
        "info": {
            "directory": "onelake://your-workspace-name/your-lakehouse-name/",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    },
    "fabric_lakehouse": {
        "ds_type": "fabric",
        "auth_mode": "service_principal",
        "fabric_target": "lakehouse",
        "info": {
            "workspace_id": "your-workspace-id",
            "lakehouse_id": "your-lakehouse-id",
            "lakehouse_name": "your-lakehouse-name",
            "sql_endpoint": "your-sql-endpoint.datawarehouse.fabric.microsoft.com",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    }
}
```

**Configuration Fields:**

| Field                 | Description                                                                                          | Required                                                                                                                                                               |
| --------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ds_type`             | Must be `"fabric"`                                                                                   | Yes                                                                                                                                                                    |
| `auth_mode`           | Must be `"service_principal"`                                                                        | Yes                                                                                                                                                                    |
| `fabric_target`       | Set to `"lakehouse"` to use the Lakehouse (Delta tables / OneLake) publisher documented on this page | No -- if omitted (or set to anything other than `"lakehouse"`), LakeXpress instead targets a Fabric **Warehouse** (T-SQL) publisher, which is not covered by this page |
| `workspace_id`        | Fabric workspace GUID                                                                                | Yes                                                                                                                                                                    |
| `lakehouse_id`        | Lakehouse GUID                                                                                       | Yes                                                                                                                                                                    |
| `lakehouse_name`      | Lakehouse display name                                                                               | Yes                                                                                                                                                                    |
| `sql_endpoint`        | SQL analytics endpoint hostname                                                                      | Yes                                                                                                                                                                    |
| `azure_client_id`     | Application (client) ID                                                                              | Yes                                                                                                                                                                    |
| `azure_tenant_id`     | Directory (tenant) ID                                                                                | Yes                                                                                                                                                                    |
| `azure_client_secret` | Client secret value                                                                                  | Yes                                                                                                                                                                    |

## Authentication Setup[​](#authentication-setup "Direct link to Authentication Setup")

Fabric authenticates via Service Principal (Azure AD).

### Service Principal Authentication[​](#service-principal-authentication "Direct link to Service Principal Authentication")

```
{
    "fabric_lakehouse": {
        "ds_type": "fabric",
        "auth_mode": "service_principal",
        "fabric_target": "lakehouse",
        "info": {
            "workspace_id": "your-workspace-id",
            "lakehouse_id": "your-lakehouse-id",
            "lakehouse_name": "your-lakehouse-name",
            "sql_endpoint": "your-sql-endpoint.datawarehouse.fabric.microsoft.com",
            "azure_client_id": "your-application-client-id",
            "azure_tenant_id": "your-directory-tenant-id",
            "azure_client_secret": "your-client-secret"
        }
    }
}
```

### OneLake Directory Format[​](#onelake-directory-format "Direct link to OneLake Directory Format")

The OneLake directory must match your Fabric workspace and Lakehouse:

```
onelake://workspace-name/lakehouse-name/
```

Or using the Files path:

```
onelake://workspace-name/lakehouse-name.Lakehouse/Files/
```

## Table Types[​](#table-types "Direct link to Table Types")

LakeXpress supports two Fabric Lakehouse table modes.

### Internal Tables (Managed Delta Tables) - Recommended[​](#internal-tables-managed-delta-tables---recommended "Direct link to Internal Tables (Managed Delta Tables) - Recommended")

Delta tables in the Lakehouse Tables section. Data is converted to Delta Lake format with ACID transactions, time travel, and automatic SQL analytics endpoint availability.

* Full Delta Lake features (time travel, ACID, versioning)
* Optimized query performance
* Native Power BI integration
* Best for production workloads

Options for `config create`:

```
--publish_target fabric_lakehouse
--publish_method internal
```

### External Tables (SQL Analytics Views)[​](#external-tables-sql-analytics-views "Direct link to External Tables (SQL Analytics Views)")

SQL views referencing Parquet files in the Files section. Data stays as Parquet; queries read files directly. Available via the SQL analytics endpoint only.

* No data conversion overhead
* Available immediately after export
* Suited for exploration and ad-hoc analysis
* Preserves original Parquet format

Options for `config create`:

```
--publish_target fabric_lakehouse
--publish_method external
```

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                                                                                                                                                                                                                                    | Default                                                         |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `--publish_target ID`              | Credential ID for Fabric target (required)                                                                                                                                                                                                                     | -                                                               |
| `--publish_method METHOD`          | `internal` (Delta tables, recommended) or `external` (SQL views)                                                                                                                                                                                               | `internal`                                                      |
| `--publish_schema_pattern PATTERN` | Not applicable for the Lakehouse target -- the Lakehouse has a flat namespace, so schema and table are combined via `--publish_table_pattern`                                                                                                                  | -                                                               |
| `--publish_table_pattern PATTERN`  | Table naming pattern. The Lakehouse namespace is flat: with several source schemas, include `{schema}` (e.g. `{schema}_{table}`) so tables with the same name do not collide. To publish views next to tables of the same name, use e.g. `VW_{schema}_{table}` | `{schema}_{table}` (internal), `VW_{schema}_{table}` (external) |
| `--n_jobs N`                       | Parallel workers for table creation                                                                                                                                                                                                                            | `1`                                                             |

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token      | Description        | Example Output |
| ---------- | ------------------ | -------------- |
| `{schema}` | Source schema name | `tpch_1`       |
| `{table}`  | Source table name  | `customer`     |

The Lakehouse target substitutes only `{schema}` and `{table}`. Other tokens (`{date}`, `{timestamp}`, `{uuid}`, `{subpath}`, `{database}`) are not replaced: their braces become `_` when LakeXpress sanitizes the name, which replaces every character other than letters, digits and `_`.

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Prefixed Tables

```
--publish_table_pattern "src_{schema}_{table}"
--publish_target fabric_lakehouse
--publish_method internal

# Result: src_tpch_1_customer, src_tpch_1_orders
```

Schema-Based Naming

```
--publish_table_pattern "{schema}_{table}"
--publish_target fabric_lakehouse
--publish_method internal

# Result: tpch_1_customer, tpch_1_orders
```

SQL Views with Prefix

```
--publish_table_pattern "VW_{schema}_{table}"
--publish_target fabric_lakehouse
--publish_method external

# Result: VW_tpch_1_customer, VW_tpch_1_orders
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for Microsoft Fabric are in [Examples & Recipes: Microsoft Fabric](/latest/guides/recipes/fabric.md).

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

The Lakehouse target supports only the `append` strategy. `upsert` and `delete_insert` are available with the Fabric Warehouse target (`fabric_target` omitted). See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Reference](/latest/integrations/fabric-reference.md)Data type mappings, querying, troubleshooting, CLI arguments
* [![](/img/logos/onelake.png)Intermediate Storage](/latest/configuration/storage/onelake.md)OneLake configuration
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# Microsoft Fabric Reference

Reference material for [Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md) -- data type mappings, querying, troubleshooting, and CLI arguments.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

These mappings apply to the Fabric **Warehouse** target (`fabric_target` omitted), whose tables LakeXpress creates with T-SQL DDL. With the Lakehouse target (`"fabric_target": "lakehouse"`), Delta tables and SQL views take their column types from the Parquet files, as written by FastBCP.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to Fabric[​](#postgresql-to-fabric "Direct link to PostgreSQL to Fabric")

| PostgreSQL Type              | Fabric Type        |
| ---------------------------- | ------------------ |
| `SMALLINT`                   | `SMALLINT`         |
| `INTEGER`, `INT4`            | `INT`              |
| `BIGINT`, `INT8`             | `BIGINT`           |
| `NUMERIC(p,s)`               | `DECIMAL(p,s)`     |
| `REAL`, `FLOAT4`             | `REAL`             |
| `DOUBLE PRECISION`, `FLOAT8` | `FLOAT`            |
| `VARCHAR(n)`                 | `VARCHAR(n)`       |
| `TEXT`                       | `VARCHAR(MAX)`     |
| `CHAR(n)`                    | `CHAR(n)`          |
| `BOOLEAN`                    | `BIT`              |
| `DATE`                       | `DATE`             |
| `TIME`                       | `TIME`             |
| `TIMESTAMP`                  | `DATETIME2`        |
| `TIMESTAMPTZ`                | `DATETIMEOFFSET`   |
| `BYTEA`                      | `VARBINARY(MAX)`   |
| `UUID`                       | `UNIQUEIDENTIFIER` |
| `JSON`, `JSONB`, `array []`  | `NVARCHAR(MAX)`    |

### SQL Server to Fabric[​](#sql-server-to-fabric "Direct link to SQL Server to Fabric")

| SQL Server Type                          | Fabric Type        |
| ---------------------------------------- | ------------------ |
| `TINYINT`                                | `TINYINT`          |
| `SMALLINT`                               | `SMALLINT`         |
| `INT`                                    | `INT`              |
| `BIGINT`                                 | `BIGINT`           |
| `DECIMAL(p,s)`                           | `DECIMAL(p,s)`     |
| `MONEY`                                  | `DECIMAL(19,4)`    |
| `SMALLMONEY`                             | `DECIMAL(10,4)`    |
| `FLOAT`                                  | `FLOAT`            |
| `REAL`                                   | `REAL`             |
| `BIT`                                    | `BIT`              |
| `VARCHAR(n)`                             | `VARCHAR(n)`       |
| `NVARCHAR(n)`                            | `NVARCHAR(n)`      |
| `CHAR(n)`                                | `CHAR(n)`          |
| `NCHAR(n)`                               | `NCHAR(n)`         |
| `TEXT`                                   | `VARCHAR(MAX)`     |
| `NTEXT`                                  | `NVARCHAR(MAX)`    |
| `DATE`                                   | `DATE`             |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME` | `DATETIME2`        |
| `DATETIMEOFFSET`                         | `DATETIMEOFFSET`   |
| `VARBINARY`                              | `VARBINARY(MAX)`   |
| `UNIQUEIDENTIFIER`                       | `UNIQUEIDENTIFIER` |

### Oracle to Fabric[​](#oracle-to-fabric "Direct link to Oracle to Fabric")

| Oracle Type    | Fabric Type      |
| -------------- | ---------------- |
| `NUMBER`       | `DECIMAL(38,18)` |
| `NUMBER(p,s)`  | `DECIMAL(p,s)`   |
| `FLOAT`        | `FLOAT`          |
| `VARCHAR2(n)`  | `VARCHAR(n)`     |
| `NVARCHAR2(n)` | `NVARCHAR(n)`    |
| `CHAR(n)`      | `CHAR(n)`        |
| `CLOB`         | `VARCHAR(MAX)`   |
| `NCLOB`        | `NVARCHAR(MAX)`  |
| `DATE`         | `DATE`           |
| `TIMESTAMP`    | `DATETIME2`      |
| `RAW`, `BLOB`  | `VARBINARY(MAX)` |

### Teradata to Fabric[​](#teradata-to-fabric "Direct link to Teradata to Fabric")

| Teradata Type                  | Fabric Type      |
| ------------------------------ | ---------------- |
| `BYTEINT`, `VARBYTE`           | `NVARCHAR(MAX)`  |
| `SMALLINT`                     | `SMALLINT`       |
| `INTEGER`                      | `INT`            |
| `BIGINT`                       | `BIGINT`         |
| `DECIMAL(p,s)`, `NUMERIC(p,s)` | `DECIMAL(p,s)`   |
| `NUMBER`                       | `DECIMAL(38,18)` |
| `FLOAT`, `REAL`                | `FLOAT`          |
| `VARCHAR(n)`                   | `VARCHAR(n)`     |
| `CHAR(n)`                      | `CHAR(n)`        |
| `CLOB`                         | `VARCHAR(MAX)`   |
| `DATE`                         | `DATE`           |
| `TIME`                         | `TIME`           |
| `TIMESTAMP`                    | `DATETIME2`      |
| `BYTE`, `BLOB`                 | `VARBINARY(MAX)` |

### MySQL to Fabric[​](#mysql-to-fabric "Direct link to MySQL to Fabric")

| MySQL Type                                                                                                             | Fabric Type      |
| ---------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `TINYINT`                                                                                                              | `TINYINT`        |
| `SMALLINT`                                                                                                             | `SMALLINT`       |
| `MEDIUMINT`, `YEAR`, `TINYTEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`, `JSON` | `NVARCHAR(MAX)`  |
| `INT`                                                                                                                  | `INT`            |
| `BIGINT`                                                                                                               | `BIGINT`         |
| `DECIMAL(p,s)`                                                                                                         | `DECIMAL(p,s)`   |
| `FLOAT`, `DOUBLE`                                                                                                      | `FLOAT`          |
| `BIT`                                                                                                                  | `BIT`            |
| `VARCHAR(n)`                                                                                                           | `VARCHAR(n)`     |
| `CHAR(n)`                                                                                                              | `CHAR(n)`        |
| `TEXT`                                                                                                                 | `VARCHAR(MAX)`   |
| `DATE`                                                                                                                 | `DATE`           |
| `TIME`                                                                                                                 | `TIME`           |
| `DATETIME`, `TIMESTAMP`                                                                                                | `DATETIME2`      |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                          | `VARBINARY(MAX)` |

### MariaDB to Fabric[​](#mariadb-to-fabric "Direct link to MariaDB to Fabric")

| MariaDB Type                                                                                                           | Fabric Type      |
| ---------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `TINYINT`                                                                                                              | `TINYINT`        |
| `SMALLINT`                                                                                                             | `SMALLINT`       |
| `MEDIUMINT`, `YEAR`, `TINYTEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`, `JSON` | `NVARCHAR(MAX)`  |
| `INT`                                                                                                                  | `INT`            |
| `BIGINT`                                                                                                               | `BIGINT`         |
| `DECIMAL(p,s)`                                                                                                         | `DECIMAL(p,s)`   |
| `FLOAT`, `DOUBLE`                                                                                                      | `FLOAT`          |
| `BIT`                                                                                                                  | `BIT`            |
| `VARCHAR(n)`                                                                                                           | `VARCHAR(n)`     |
| `CHAR(n)`                                                                                                              | `CHAR(n)`        |
| `TEXT`                                                                                                                 | `VARCHAR(MAX)`   |
| `DATE`                                                                                                                 | `DATE`           |
| `TIME`                                                                                                                 | `TIME`           |
| `DATETIME`, `TIMESTAMP`                                                                                                | `DATETIME2`      |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                          | `VARBINARY(MAX)` |

### SAP HANA to Fabric[​](#sap-hana-to-fabric "Direct link to SAP HANA to Fabric")

| SAP HANA Type                         | Fabric Type      |
| ------------------------------------- | ---------------- |
| `TINYINT`                             | `TINYINT`        |
| `SMALLINT`                            | `SMALLINT`       |
| `INTEGER`                             | `INT`            |
| `BIGINT`                              | `BIGINT`         |
| `DECIMAL(p,s)`                        | `DECIMAL(p,s)`   |
| `SMALLDECIMAL`, `NCLOB`, `SECONDDATE` | `NVARCHAR(MAX)`  |
| `REAL`                                | `REAL`           |
| `DOUBLE`                              | `FLOAT`          |
| `BOOLEAN`                             | `BIT`            |
| `VARCHAR(n)`                          | `VARCHAR(n)`     |
| `NVARCHAR(n)`                         | `NVARCHAR(n)`    |
| `CHAR(n)`                             | `CHAR(n)`        |
| `NCHAR(n)`                            | `NCHAR(n)`       |
| `CLOB`, `TEXT`                        | `VARCHAR(MAX)`   |
| `DATE`                                | `DATE`           |
| `TIME`                                | `TIME`           |
| `TIMESTAMP`                           | `DATETIME2`      |
| `VARBINARY`, `BLOB`                   | `VARBINARY(MAX)` |

## Querying Tables[​](#querying-tables "Direct link to Querying Tables")

### SQL Analytics Endpoint[​](#sql-analytics-endpoint "Direct link to SQL Analytics Endpoint")

```
-- Query a Delta table (default name: {schema}_{table})
SELECT TOP 10 * FROM your_lakehouse.dbo.tpch_1_customer;

-- Aggregation
SELECT
    c_nationkey,
    COUNT(*) as customer_count,
    SUM(c_acctbal) as total_balance
FROM your_lakehouse.dbo.tpch_1_customer
GROUP BY c_nationkey
ORDER BY customer_count DESC;
```

### Spark Notebooks[​](#spark-notebooks "Direct link to Spark Notebooks")

```
# Read a Delta table
df = spark.read.table("tpch_1_customer")
df.show(10)

# Query with SQL
spark.sql("""
    SELECT c_nationkey, COUNT(*) as cnt
    FROM tpch_1_customer
    GROUP BY c_nationkey
""").show()

# Write results back
result_df.write.mode("overwrite").saveAsTable("customer_summary")
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

#### Authentication Errors[​](#authentication-errors "Direct link to Authentication Errors")

**Error:** `AADSTS7000215: Invalid client secret provided`

Regenerate your client secret in Azure AD and update `credentials.json`.

**Error:** `The user or service principal does not have access to the workspace`

1. Verify the Service Principal is added to the workspace
2. Ensure it has Member or Contributor role
3. Wait a few minutes for permissions to propagate

#### OneLake Connection Issues[​](#onelake-connection-issues "Direct link to OneLake Connection Issues")

**Error:** `Unable to connect to OneLake storage`

1. Verify the directory format: `onelake://workspace-name/lakehouse-name/`
2. Check that the Service Principal has Storage Blob Data Contributor role
3. Ensure workspace and lakehouse names are correct (case-sensitive)

#### Table Creation Failures[​](#table-creation-failures "Direct link to Table Creation Failures")

**Error:** `Failed to create table in Lakehouse`

1. Verify the SQL analytics endpoint is correct
2. Check that the Lakehouse is active (not paused)
3. Ensure sufficient capacity units are available
4. Confirm Parquet files were exported to OneLake

#### SQL Endpoint Connection Issues[​](#sql-endpoint-connection-issues "Direct link to SQL Endpoint Connection Issues")

**Error:** `Cannot connect to SQL analytics endpoint`

1. Verify the SQL endpoint hostname
2. Ensure the SQL analytics endpoint is enabled
3. Check firewall rules if connecting from on-premises

### Verifying Export Success[​](#verifying-export-success "Direct link to Verifying Export Success")

Check files in OneLake:

1. Open your Lakehouse in Fabric portal
2. Go to the **Files** section
3. Verify Parquet files exist in the expected folders

Check tables:

1. Open the **SQL analytics endpoint**
2. Expand **Tables** in the object explorer
3. Verify your tables appear

### Debug Mode[​](#debug-mode "Direct link to Debug Mode")

```
./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID> \
    --log_level DEBUG
```

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Fabric Publishing Arguments[​](#fabric-publishing-arguments "Direct link to Fabric Publishing Arguments")

| Option                            | Type    | Description                                                                                                                                                                                                         |
| --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--publish_target ID`             | String  | Credential ID for Fabric publishing (required)                                                                                                                                                                      |
| `--publish_method METHOD`         | Enum    | `internal` (Delta tables, recommended) or `external` (SQL views) (default: `internal`)                                                                                                                              |
| `--publish_table_pattern PATTERN` | String  | Table naming pattern for the Lakehouse's flat namespace (default: `{schema}_{table}` for internal, `VW_{schema}_{table}` for external; include `{schema}`, e.g. `{schema}_{table}`, when exporting several schemas) |
| `--n_jobs N`                      | Integer | Parallel workers for table creation (default: 1)                                                                                                                                                                    |

### Full Example[​](#full-example "Direct link to Full Example")

See [Examples & Recipes: Microsoft Fabric](/latest/guides/recipes/fabric.md) for complete, ready-to-run commands.

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md)Setup and usage guide
* [![](/img/logos/onelake.png)Intermediate Storage](/latest/configuration/storage/onelake.md)OneLake configuration
* [CLI Reference](/latest/cli/.md)All command-line options


---

# AWS Glue Publishing

LakeXpress creates **AWS Glue Data Catalog tables** from exported Parquet files, enabling queries via Athena, Redshift Spectrum, and EMR.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. AWS Glue Permissions[​](#1-aws-glue-permissions "Direct link to 1. AWS Glue Permissions")

Required IAM permissions:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateDatabase",
                "glue:GetDatabase",
                "glue:DeleteDatabase",
                "glue:CreateTable",
                "glue:GetTable",
                "glue:DeleteTable",
                "glue:UpdateTable",
                "glue:GetTables",
                "glue:BatchCreatePartition",
                "glue:GetPartitions"
            ],
            "Resource": "*"
        }
    ]
}
```

### 2. S3 Storage Setup[​](#2-s3-storage-setup "Direct link to 2. S3 Storage Setup")

Both S3 and Glue credentials are required in `credentials.json`:

```
{
    "s3_01": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://my-datalake-bucket/lakexpress/",
            "profile": "my-aws-profile"
        }
    },
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}
```

tip

Both `"aws_glue"` and `"glue"` are accepted as the `ds_type` value.

**Critical:** The S3 bucket must be accessible from the Glue Data Catalog in the specified region.

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

* Profile (Recommended)
* Keys
* Role

Uses AWS CLI credentials from `~/.aws/credentials`:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}
```

* No secrets in config files
* Supports MFA and SSO profiles
* Easy credential rotation

Uses explicit AWS access keys:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "keys",
        "info": {
            "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
            "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "region": "us-east-1"
        }
    }
}
```

Suited for CI/CD pipelines, containers, or cross-account access without AWS CLI.

Assumes an IAM role via STS:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "role",
        "info": {
            "role_arn": "arn:aws:iam::123456789012:role/GluePublishRole",
            "external_id": "optional-external-id",
            "region": "us-east-1"
        }
    }
}
```

Uses temporary credentials. Ideal for cross-account publishing and fine-grained access control.

### Optional Field: `catalog_id`[​](#optional-field-catalog_id "Direct link to optional-field-catalog_id")

All three `auth_mode` values above also accept an optional `catalog_id` field in `info`, to target a Data Catalog in a different AWS account than the one whose credentials are used:

```
{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1",
            "catalog_id": "123456789012"
        }
    }
}
```

If omitted, `catalog_id` defaults to the caller's own AWS account.

## Understanding Glue Database Naming[​](#understanding-glue-database-naming "Direct link to Understanding Glue Database Naming")

Two ways to set the Glue database name

Unlike traditional databases, the AWS Glue Data Catalog has no server or database instance to connect to. What Glue calls a "database" is simply a **namespace** — a container for organizing tables within the catalog. The catalog itself (scoped by `catalog_id`, which defaults to your AWS account) is the top-level container.

By default, `--publish_schema_pattern` controls the **Glue database name** that will be created. For example, `--publish_schema_pattern "datalake_{schema}"` creates a Glue database called `datalake_tpch_1`.

`--publish_database_name` also exists and, when set, **takes precedence** over `--publish_schema_pattern` for Glue: tokens are substituted as in `--publish_schema_pattern`, and the name is lowercased like every Glue database name.

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                                                                        | Default        |
| ---------------------------------- | -------------------------------------------------------------------------------------------------- | -------------- |
| `--publish_target ID`              | Credential ID for Glue target (required)                                                           | -              |
| `--publish_schema_pattern PATTERN` | Glue database naming pattern                                                                       | `EXT_{schema}` |
| `--publish_database_name NAME`     | Glue database name (tokens substituted, lowercased); overrides `--publish_schema_pattern` when set | -              |
| `--publish_table_pattern PATTERN`  | Table naming pattern                                                                               | `{table}`      |
| `--n_jobs N`                       | Parallel workers for table creation                                                                | `1`            |

note

There is no `--glue_skip_existing` CLI flag. AWS Glue publishing always (re)creates tables; existing-table skip behavior is not exposed on the command line.

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Database and table names support token-based patterns.

Token restrictions

The `{table}` token can only be used in `--publish_table_pattern`, not in `--publish_schema_pattern`. Since the schema pattern controls the Glue database name, using `{table}` there would attempt to create a separate database per table, which is not supported. LakeXpress validates this before starting the export and will report an error.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                          | Example Output    |
| ------------- | ------------------------------------ | ----------------- |
| `{schema}`    | Source schema name                   | `tpch_1`          |
| `{table}`     | Source table name                    | `customer`        |
| `{date}`      | Current date (YYYYMMDD)              | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS) | `20251210_143022` |
| `{uuid}`      | UUID4 (consistent per run)           | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` value               | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Date-Partitioned Databases

```
--publish_schema_pattern "lx_{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target glue_catalog

# Results:
# Database: lx_tpch_1_20251210
# Tables: customer, orders, lineitem
```

Prefixed Databases

```
--publish_schema_pattern "datalake_{schema}"
--publish_table_pattern "{table}"
--publish_target glue_catalog

# Results:
# Database: datalake_tpch_1
# Tables: customer, orders, lineitem
```

Consolidated Multi-Schema

```
--source_schema_name schema1,schema2
--publish_schema_pattern "consolidated"
--publish_table_pattern "{schema}_{table}"
--publish_target glue_catalog

# Results:
# Database: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for AWS Glue are in [Examples & Recipes: AWS Glue](/latest/guides/recipes/glue.md).

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

AWS Glue supports only the `append` strategy: `upsert` and `delete_insert` stop the publish step with an error. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/aws-glue.svg)AWS Glue Reference](/latest/integrations/glue-reference.md)Data type mappings, CLI arguments, querying
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# AWS Glue Reference

Reference material for [AWS Glue Publishing](/latest/integrations/glue-publishing.md) -- data type mappings, CLI arguments, and querying.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to Glue-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to Glue[​](#postgresql-to-glue "Direct link to PostgreSQL to Glue")

| PostgreSQL Type                                                              | Glue Type      |
| ---------------------------------------------------------------------------- | -------------- |
| `SMALLINT`                                                                   | `smallint`     |
| `INTEGER`, `INT4`                                                            | `int`          |
| `BIGINT`, `INT8`                                                             | `bigint`       |
| `NUMERIC(p,s)`                                                               | `decimal(p,s)` |
| `REAL`, `FLOAT4`                                                             | `float`        |
| `DOUBLE PRECISION`, `FLOAT8`                                                 | `double`       |
| `VARCHAR(n)`, `TEXT`, `CHAR(n)`, `TIME`, `UUID`, `JSON`, `JSONB`, `array []` | `string`       |
| `BOOLEAN`                                                                    | `boolean`      |
| `DATE`                                                                       | `date`         |
| `TIMESTAMP`, `TIMESTAMPTZ`                                                   | `timestamp`    |
| `BYTEA`                                                                      | `binary`       |

### SQL Server to Glue[​](#sql-server-to-glue "Direct link to SQL Server to Glue")

| SQL Server Type                                                                         | Glue Type       |
| --------------------------------------------------------------------------------------- | --------------- |
| `TINYINT`                                                                               | `tinyint`       |
| `SMALLINT`                                                                              | `smallint`      |
| `INT`                                                                                   | `int`           |
| `BIGINT`                                                                                | `bigint`        |
| `DECIMAL(p,s)`                                                                          | `decimal(p,s)`  |
| `MONEY`                                                                                 | `decimal(19,4)` |
| `SMALLMONEY`                                                                            | `decimal(10,4)` |
| `FLOAT`, `REAL`                                                                         | `float`         |
| `BIT`                                                                                   | `boolean`       |
| `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `TEXT`, `NTEXT`, `UNIQUEIDENTIFIER` | `string`        |
| `DATE`                                                                                  | `date`          |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME`, `DATETIMEOFFSET`                              | `timestamp`     |
| `VARBINARY`                                                                             | `binary`        |

### Oracle to Glue[​](#oracle-to-glue "Direct link to Oracle to Glue")

| Oracle Type                                               | Glue Type        |
| --------------------------------------------------------- | ---------------- |
| `NUMBER`                                                  | `decimal(38,18)` |
| `NUMBER(p,s)`                                             | `decimal(p,s)`   |
| `FLOAT`                                                   | `float`          |
| `VARCHAR2(n)`, `NVARCHAR2(n)`, `CHAR(n)`, `CLOB`, `NCLOB` | `string`         |
| `DATE`                                                    | `date`           |
| `TIMESTAMP`                                               | `timestamp`      |
| `RAW`, `BLOB`                                             | `binary`         |

### Teradata to Glue[​](#teradata-to-glue "Direct link to Teradata to Glue")

| Teradata Type                                                 | Glue Type        |
| ------------------------------------------------------------- | ---------------- |
| `BYTEINT`, `VARCHAR(n)`, `CHAR(n)`, `CLOB`, `TIME`, `VARBYTE` | `string`         |
| `SMALLINT`                                                    | `smallint`       |
| `INTEGER`                                                     | `int`            |
| `BIGINT`                                                      | `bigint`         |
| `DECIMAL(p,s)`, `NUMERIC(p,s)`                                | `decimal(p,s)`   |
| `NUMBER`                                                      | `decimal(38,18)` |
| `FLOAT`, `REAL`                                               | `float`          |
| `DATE`                                                        | `date`           |
| `TIMESTAMP`                                                   | `timestamp`      |
| `BYTE`, `BLOB`                                                | `binary`         |

### MySQL to Glue[​](#mysql-to-glue "Direct link to MySQL to Glue")

| MySQL Type                                                                                                                                                      | Glue Type      |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- |
| `TINYINT`                                                                                                                                                       | `tinyint`      |
| `SMALLINT`                                                                                                                                                      | `smallint`     |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TIME`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`, `JSON` | `string`       |
| `INT`                                                                                                                                                           | `int`          |
| `BIGINT`                                                                                                                                                        | `bigint`       |
| `DECIMAL(p,s)`                                                                                                                                                  | `decimal(p,s)` |
| `FLOAT`                                                                                                                                                         | `float`        |
| `DOUBLE`                                                                                                                                                        | `double`       |
| `BIT`                                                                                                                                                           | `boolean`      |
| `DATE`                                                                                                                                                          | `date`         |
| `DATETIME`, `TIMESTAMP`                                                                                                                                         | `timestamp`    |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                                   | `binary`       |

### MariaDB to Glue[​](#mariadb-to-glue "Direct link to MariaDB to Glue")

| MariaDB Type                                                                                                                                                    | Glue Type      |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- |
| `TINYINT`                                                                                                                                                       | `tinyint`      |
| `SMALLINT`                                                                                                                                                      | `smallint`     |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TIME`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`, `JSON` | `string`       |
| `INT`                                                                                                                                                           | `int`          |
| `BIGINT`                                                                                                                                                        | `bigint`       |
| `DECIMAL(p,s)`                                                                                                                                                  | `decimal(p,s)` |
| `FLOAT`                                                                                                                                                         | `float`        |
| `DOUBLE`                                                                                                                                                        | `double`       |
| `BIT`                                                                                                                                                           | `boolean`      |
| `DATE`                                                                                                                                                          | `date`         |
| `DATETIME`, `TIMESTAMP`                                                                                                                                         | `timestamp`    |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                                   | `binary`       |

### SAP HANA to Glue[​](#sap-hana-to-glue "Direct link to SAP HANA to Glue")

| SAP HANA Type                                                                                                     | Glue Type      |
| ----------------------------------------------------------------------------------------------------------------- | -------------- |
| `TINYINT`                                                                                                         | `tinyint`      |
| `SMALLINT`                                                                                                        | `smallint`     |
| `INTEGER`                                                                                                         | `int`          |
| `BIGINT`                                                                                                          | `bigint`       |
| `DECIMAL(p,s)`                                                                                                    | `decimal(p,s)` |
| `SMALLDECIMAL`, `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `CLOB`, `NCLOB`, `TEXT`, `TIME`, `SECONDDATE` | `string`       |
| `REAL`                                                                                                            | `float`        |
| `DOUBLE`                                                                                                          | `double`       |
| `BOOLEAN`                                                                                                         | `boolean`      |
| `DATE`                                                                                                            | `date`         |
| `TIMESTAMP`                                                                                                       | `timestamp`    |
| `VARBINARY`, `BLOB`                                                                                               | `binary`       |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Glue Publishing Arguments[​](#glue-publishing-arguments "Direct link to Glue Publishing Arguments")

| Option                             | Type    | Description                                                                                        |
| ---------------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for Glue publishing (required)                                                       |
| `--publish_schema_pattern PATTERN` | String  | Glue database naming pattern (default: `EXT_{schema}`)                                             |
| `--publish_database_name NAME`     | String  | Glue database name (tokens substituted, lowercased); overrides `--publish_schema_pattern` when set |
| `--publish_table_pattern PATTERN`  | String  | Table naming pattern (default: `{table}`)                                                          |
| `--n_jobs N`                       | Integer | Parallel workers for table creation (default: 1)                                                   |

There is no `--glue_skip_existing` flag -- AWS Glue publishing always (re)creates tables.

### Querying Glue Tables[​](#querying-glue-tables "Direct link to Querying Glue Tables")

**Amazon Athena:**

```
SELECT * FROM lx_tpch_1.customer LIMIT 10;
```

**AWS Redshift Spectrum:**

```
SELECT * FROM spectrum_schema.customer LIMIT 10;
```

**Amazon EMR (Spark):**

```
df = spark.table("lx_tpch_1.customer")
df.show(10)
```

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/aws-glue.svg)AWS Glue Publishing](/latest/integrations/glue-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options


---

# MotherDuck Publishing

LakeXpress creates MotherDuck tables from exported Parquet files in cloud storage (S3, GCS, Azure).

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. MotherDuck Account[​](#1-motherduck-account "Direct link to 1. MotherDuck Account")

Sign up at [motherduck.com](https://motherduck.com) and generate an access token:

1. Log in to MotherDuck
2. Go to Settings → Access Tokens
3. Create a new token
4. Copy the token for your credentials file

### 2. Cloud Storage Setup[​](#2-cloud-storage-setup "Direct link to 2. Cloud Storage Setup")

Configure credentials for cloud storage and MotherDuck:

```
{
    "s3_01": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://my-datalake-bucket/lakexpress/",
            "profile": "my-aws-profile"
        }
    },
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-access-token"
        }
    }
}
```

### 3. Storage Access from MotherDuck (Automatic, with a Manual Fallback)[​](#3-storage-access-from-motherduck-automatic-with-a-manual-fallback "Direct link to 3. Storage Access from MotherDuck (Automatic, with a Manual Fallback)")

MotherDuck needs credentials to read your Parquet files from cloud storage, whether you publish `external` views (queried live from storage) or `internal` tables (loaded once at publish time).

**LakeXpress configures this automatically.** On every publish, it takes the storage credential referenced by `--target_storage_id` (S3, GCS, or Azure) and issues a `CREATE OR REPLACE SECRET lakexpress_storage (...)` statement against the MotherDuck connection before creating tables/views, so no manual step is normally required.

If you still see errors like the ones below, it usually means the storage credential itself is misconfigured (e.g. a profile MotherDuck's environment cannot resolve), or a pre-existing secret is conflicting with the auto-created one:

```
IO Error: No files found that match the pattern "s3://..."
HTTP Error: Permission error: Missing or invalid credentials
```

**You can also configure the secret manually in MotherDuck** (useful for querying published views outside of a LakeXpress run, or for troubleshooting):

**For AWS S3 (most common):**

```
CREATE SECRET aws_s3_secret (
    TYPE S3,
    KEY_ID 'your_aws_access_key_id',
    SECRET 'your_aws_secret_access_key',
    REGION 'us-east-1'  -- your bucket's region
);
```

To find your AWS credentials:

```
aws configure get aws_access_key_id
aws configure get aws_secret_access_key
aws configure get region
```

**For GCS:**

```
CREATE SECRET gcs_secret (
    TYPE GCS,
    KEY_ID 'your_hmac_access_key',
    SECRET 'your_hmac_secret_key'
);
```

**For Azure:**

```
CREATE SECRET azure_secret (
    TYPE AZURE,
    ACCOUNT_NAME 'your_storage_account',
    ACCOUNT_KEY 'your_account_key'
);
```

**Verify secrets are configured:**

```
SELECT * FROM duckdb_secrets();
```

## Authentication[​](#authentication "Direct link to Authentication")

* Token (Recommended)
* Environment Variable

Store your token in the credentials file:

```
{
    "motherduck_prod": {
        "ds_type": "motherduck",
        "auth_mode": "token",
        "info": {
            "database": "my_analytics",
            "token": "your-motherduck-access-token"
        }
    }
}
```

| Field       | Required | Description                     |
| ----------- | -------- | ------------------------------- |
| `ds_type`   | Yes      | Must be `"motherduck"`          |
| `auth_mode` | Yes      | Set to `"token"`                |
| `database`  | Yes      | Target MotherDuck database name |
| `token`     | Yes      | MotherDuck access token         |

Read the token from an environment variable:

```
{
    "motherduck_env": {
        "ds_type": "motherduck",
        "auth_mode": "env",
        "info": {
            "database": "my_analytics"
        }
    }
}
```

```
export motherduck_token="your-motherduck-access-token"
./LakeXpress sync ...
```

| Field       | Required | Description                     |
| ----------- | -------- | ------------------------------- |
| `ds_type`   | Yes      | Must be `"motherduck"`          |
| `auth_mode` | Yes      | Set to `"env"`                  |
| `database`  | Yes      | Target MotherDuck database name |

## Configuration Options[​](#configuration-options "Direct link to Configuration Options")

| Option                             | Description                                                                                                          | Default                                                                                             |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `--publish_target ID`              | Credential ID for MotherDuck target (required)                                                                       | -                                                                                                   |
| `--publish_schema_pattern PATTERN` | Schema naming pattern. The resolved name is lowercased, and characters other than letters, digits and `_` become `_` | `{schema}` for `internal`, `EXT_{schema}` for the default `external` method (unless set explicitly) |
| `--publish_table_pattern PATTERN`  | Table naming pattern                                                                                                 | `{table}`                                                                                           |
| `--publish_method TYPE`            | Table type: `external` or `internal`                                                                                 | `external`                                                                                          |
| `--pk_constraints`                 | Add `PRIMARY KEY` constraints to native (`internal`) table DDL; not supported for `external` views                   | off                                                                                                 |
| `--n_jobs N`                       | Parallel workers for table creation                                                                                  | `1`                                                                                                 |

## Table Types[​](#table-types "Direct link to Table Types")

### External Views (Default)[​](#external-views-default "Direct link to External Views (Default)")

Queries data directly from cloud storage via `read_parquet()`.

Options for `config create`:

```
--publish_target motherduck_prod
--publish_method external
```

**Generated SQL** (default schema `EXT_{schema}`, lowercased):

```
CREATE OR REPLACE VIEW "my_database"."ext_tpch_1"."orders" AS
SELECT "o_orderkey", "o_custkey", ... FROM read_parquet('s3://bucket/exports/tpch_1/orders/*.parquet')
```

**Pros:**

* No data copying, instant publishing
* No MotherDuck storage costs
* Always reflects latest cloud storage data
* Best for large datasets, infrequent queries, exploration

**Trade-offs:**

* Query speed depends on cloud storage latency
* Requires cloud storage credentials in MotherDuck

### Internal Tables[​](#internal-tables "Direct link to Internal Tables")

Loads data into MotherDuck native columnar storage.

Options for `config create`:

```
--publish_target motherduck_prod
--publish_method internal
```

**Generated SQL** (default schema `{schema}`):

```
CREATE OR REPLACE TABLE "my_database"."tpch_1"."orders" AS
SELECT "o_orderkey", "o_custkey", ... FROM read_parquet('s3://bucket/exports/tpch_1/orders/*.parquet')
```

With `--pk_constraints`, LakeXpress creates the table with its columns and `PRIMARY KEY`, then inserts the rows.

**Pros:**

* Faster queries (optimized columnar format)
* No storage credentials needed after load
* Better caching for repeated queries
* Best for dashboards and production analytics

**Trade-offs:**

* Slower publishing (data must load)
* MotherDuck storage costs apply
* Data is a point-in-time snapshot

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                           | Example Output    |
| ------------- | ------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                    | `tpch_1`          |
| `{table}`     | Source table name                     | `customer`        |
| `{date}`      | Current date (YYYYMMDD)               | `20251210`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  | `20251210_143022` |
| `{uuid}`      | UUID4 identifier (consistent per run) | `a1b2c3d4`        |
| `{subpath}`   | CLI `--sub_path` parameter            | `staging`         |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Prefixed Schemas

```
--publish_schema_pattern "lx_{schema}"
--publish_table_pattern "{table}"
--publish_target motherduck_prod

# Results:
# Schema: lx_tpch_1
# Tables: customer, orders, lineitem
```

Date-Partitioned Schemas

```
--publish_schema_pattern "{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target motherduck_prod

# Results:
# Schema: tpch_1_20251210
# Tables: customer, orders, lineitem
```

Consolidated Multi-Schema

```
--source_schema_name schema1,schema2
--publish_schema_pattern "consolidated"
--publish_table_pattern "{schema}_{table}"
--publish_target motherduck_prod

# Results:
# Schema: consolidated
# Tables: schema1_customer, schema2_customer
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for MotherDuck are in [Examples & Recipes: MotherDuck](/latest/guides/recipes/motherduck.md).

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. A table using `upsert` or `delete_insert` needs a primary key and is always created as an internal table. MotherDuck has no `MERGE`: both strategies delete the matching rows, then insert. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/motherduck.svg)MotherDuck Reference](/latest/integrations/motherduck-reference.md)Data type mappings, CLI arguments, querying, troubleshooting
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# MotherDuck Reference

Reference material for [MotherDuck Publishing](/latest/integrations/motherduck-publishing.md) -- data type mappings, CLI arguments, querying, and troubleshooting.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to DuckDB/MotherDuck-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to DuckDB/MotherDuck[​](#postgresql-to-duckdbmotherduck "Direct link to PostgreSQL to DuckDB/MotherDuck")

| PostgreSQL Type                             | DuckDB/MotherDuck Type |
| ------------------------------------------- | ---------------------- |
| `SMALLINT`                                  | `SMALLINT`             |
| `INTEGER`, `INT4`                           | `INTEGER`              |
| `BIGINT`, `INT8`                            | `BIGINT`               |
| `NUMERIC(p,s)`                              | `DECIMAL(p,s)`         |
| `REAL`, `FLOAT4`                            | `REAL`                 |
| `DOUBLE PRECISION`, `FLOAT8`                | `DOUBLE`               |
| `VARCHAR(n)`, `TEXT`, `CHAR(n)`, `array []` | `VARCHAR`              |
| `BOOLEAN`                                   | `BOOLEAN`              |
| `DATE`                                      | `DATE`                 |
| `TIME`                                      | `TIME`                 |
| `TIMESTAMP`                                 | `TIMESTAMP`            |
| `TIMESTAMPTZ`                               | `TIMESTAMPTZ`          |
| `BYTEA`                                     | `BLOB`                 |
| `UUID`                                      | `UUID`                 |
| `JSON`, `JSONB`                             | `JSON`                 |

### SQL Server to DuckDB/MotherDuck[​](#sql-server-to-duckdbmotherduck "Direct link to SQL Server to DuckDB/MotherDuck")

| SQL Server Type                                                     | DuckDB/MotherDuck Type |
| ------------------------------------------------------------------- | ---------------------- |
| `TINYINT`                                                           | `TINYINT`              |
| `SMALLINT`                                                          | `SMALLINT`             |
| `INT`                                                               | `INTEGER`              |
| `BIGINT`                                                            | `BIGINT`               |
| `DECIMAL(p,s)`                                                      | `DECIMAL(p,s)`         |
| `MONEY`                                                             | `DECIMAL(19,4)`        |
| `SMALLMONEY`                                                        | `DECIMAL(10,4)`        |
| `FLOAT`, `REAL`                                                     | `REAL`                 |
| `BIT`                                                               | `BOOLEAN`              |
| `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `TEXT`, `NTEXT` | `VARCHAR`              |
| `DATE`                                                              | `DATE`                 |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME`                            | `TIMESTAMP`            |
| `DATETIMEOFFSET`                                                    | `TIMESTAMPTZ`          |
| `VARBINARY`                                                         | `BLOB`                 |
| `UNIQUEIDENTIFIER`                                                  | `UUID`                 |

### Oracle to DuckDB/MotherDuck[​](#oracle-to-duckdbmotherduck "Direct link to Oracle to DuckDB/MotherDuck")

| Oracle Type                                               | DuckDB/MotherDuck Type |
| --------------------------------------------------------- | ---------------------- |
| `NUMBER`                                                  | `DECIMAL(38,9)`        |
| `NUMBER(p,s)`                                             | `DECIMAL(p,s)`         |
| `FLOAT`                                                   | `REAL`                 |
| `VARCHAR2(n)`, `NVARCHAR2(n)`, `CHAR(n)`, `CLOB`, `NCLOB` | `VARCHAR`              |
| `DATE`                                                    | `DATE`                 |
| `TIMESTAMP`                                               | `TIMESTAMP`            |
| `RAW`, `BLOB`                                             | `BLOB`                 |

### Teradata to DuckDB/MotherDuck[​](#teradata-to-duckdbmotherduck "Direct link to Teradata to DuckDB/MotherDuck")

| Teradata Type                                         | DuckDB/MotherDuck Type |
| ----------------------------------------------------- | ---------------------- |
| `BYTEINT`, `VARCHAR(n)`, `CHAR(n)`, `CLOB`, `VARBYTE` | `VARCHAR`              |
| `SMALLINT`                                            | `SMALLINT`             |
| `INTEGER`                                             | `INTEGER`              |
| `BIGINT`                                              | `BIGINT`               |
| `DECIMAL(p,s)`, `NUMERIC(p,s)`                        | `DECIMAL(p,s)`         |
| `NUMBER`                                              | `DECIMAL(38,9)`        |
| `FLOAT`, `REAL`                                       | `REAL`                 |
| `DATE`                                                | `DATE`                 |
| `TIME`                                                | `TIME`                 |
| `TIMESTAMP`                                           | `TIMESTAMP`            |
| `BYTE`, `BLOB`                                        | `BLOB`                 |

### MySQL to DuckDB/MotherDuck[​](#mysql-to-duckdbmotherduck "Direct link to MySQL to DuckDB/MotherDuck")

| MySQL Type                                                                                                                                      | DuckDB/MotherDuck Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `TINYINT`                                                                                                                                       | `TINYINT`              |
| `SMALLINT`                                                                                                                                      | `SMALLINT`             |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `VARCHAR`              |
| `INT`                                                                                                                                           | `INTEGER`              |
| `BIGINT`                                                                                                                                        | `BIGINT`               |
| `DECIMAL(p,s)`                                                                                                                                  | `DECIMAL(p,s)`         |
| `FLOAT`                                                                                                                                         | `REAL`                 |
| `DOUBLE`                                                                                                                                        | `DOUBLE`               |
| `BIT`                                                                                                                                           | `BOOLEAN`              |
| `DATE`                                                                                                                                          | `DATE`                 |
| `TIME`                                                                                                                                          | `TIME`                 |
| `DATETIME`, `TIMESTAMP`                                                                                                                         | `TIMESTAMP`            |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                   | `BLOB`                 |
| `JSON`                                                                                                                                          | `JSON`                 |

### MariaDB to DuckDB/MotherDuck[​](#mariadb-to-duckdbmotherduck "Direct link to MariaDB to DuckDB/MotherDuck")

| MariaDB Type                                                                                                                                    | DuckDB/MotherDuck Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `TINYINT`                                                                                                                                       | `TINYINT`              |
| `SMALLINT`                                                                                                                                      | `SMALLINT`             |
| `MEDIUMINT`, `YEAR`, `VARCHAR(n)`, `CHAR(n)`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `VARCHAR`              |
| `INT`                                                                                                                                           | `INTEGER`              |
| `BIGINT`                                                                                                                                        | `BIGINT`               |
| `DECIMAL(p,s)`                                                                                                                                  | `DECIMAL(p,s)`         |
| `FLOAT`                                                                                                                                         | `REAL`                 |
| `DOUBLE`                                                                                                                                        | `DOUBLE`               |
| `BIT`                                                                                                                                           | `BOOLEAN`              |
| `DATE`                                                                                                                                          | `DATE`                 |
| `TIME`                                                                                                                                          | `TIME`                 |
| `DATETIME`, `TIMESTAMP`                                                                                                                         | `TIMESTAMP`            |
| `BINARY`, `VARBINARY`, `BLOB`                                                                                                                   | `BLOB`                 |
| `JSON`                                                                                                                                          | `JSON`                 |

### SAP HANA to DuckDB/MotherDuck[​](#sap-hana-to-duckdbmotherduck "Direct link to SAP HANA to DuckDB/MotherDuck")

| SAP HANA Type                                                                                             | DuckDB/MotherDuck Type |
| --------------------------------------------------------------------------------------------------------- | ---------------------- |
| `TINYINT`                                                                                                 | `TINYINT`              |
| `SMALLINT`                                                                                                | `SMALLINT`             |
| `INTEGER`                                                                                                 | `INTEGER`              |
| `BIGINT`                                                                                                  | `BIGINT`               |
| `DECIMAL(p,s)`                                                                                            | `DECIMAL(p,s)`         |
| `SMALLDECIMAL`, `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`, `CLOB`, `NCLOB`, `TEXT`, `SECONDDATE` | `VARCHAR`              |
| `REAL`                                                                                                    | `REAL`                 |
| `DOUBLE`                                                                                                  | `DOUBLE`               |
| `BOOLEAN`                                                                                                 | `BOOLEAN`              |
| `DATE`                                                                                                    | `DATE`                 |
| `TIME`                                                                                                    | `TIME`                 |
| `TIMESTAMP`                                                                                               | `TIMESTAMP`            |
| `VARBINARY`, `BLOB`                                                                                       | `BLOB`                 |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### MotherDuck Publishing Arguments[​](#motherduck-publishing-arguments "Direct link to MotherDuck Publishing Arguments")

| Option                             | Type    | Description                                                                                                                                           |
| ---------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--publish_target ID`              | String  | Credential ID for MotherDuck publishing (required)                                                                                                    |
| `--publish_schema_pattern PATTERN` | String  | Dynamic schema naming pattern (default: `{schema}` for `internal`, `EXT_{schema}` for the default `external` method; the resolved name is lowercased) |
| `--publish_table_pattern PATTERN`  | String  | Dynamic table naming pattern (default: `{table}`)                                                                                                     |
| `--publish_method TYPE`            | String  | Table type: `external` (default) or `internal`                                                                                                        |
| `--pk_constraints`                 | Flag    | Add `PRIMARY KEY` constraints to native (`internal`) table DDL; not supported for `external` views                                                    |
| `--n_jobs N`                       | Integer | Number of parallel workers for table creation (default: 1)                                                                                            |

### Querying MotherDuck Tables[​](#querying-motherduck-tables "Direct link to Querying MotherDuck Tables")

Once published, query tables via any of these methods.

**MotherDuck Web UI:**

Log in to [app.motherduck.com](https://app.motherduck.com) and run:

```
SELECT * FROM my_analytics.lx_tpch_1.customer LIMIT 10;
```

**DuckDB CLI:**

```
duckdb "md:my_analytics?motherduck_token=your_token"
```

```
SELECT * FROM lx_tpch_1.customer LIMIT 10;
```

**Python (duckdb):**

```
import duckdb

conn = duckdb.connect("md:my_analytics?motherduck_token=your_token")
df = conn.execute("SELECT * FROM lx_tpch_1.customer LIMIT 10").df()
print(df)
```

**Python with environment variable:**

```
import os
import duckdb

os.environ["motherduck_token"] = "your_token"
conn = duckdb.connect("md:my_analytics")
df = conn.execute("SELECT * FROM lx_tpch_1.customer LIMIT 10").df()
print(df)
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Common Issues[​](#common-issues "Direct link to Common Issues")

**"Authentication failed":**

* Verify your token is valid and not expired
* Check token permissions for the target database
* For env mode, ensure `motherduck_token` is set

**"Database not found":**

* Verify the database name matches an existing MotherDuck database
* Create the database in MotherDuck UI before publishing

**"IO Error: No files found that match the pattern" or "Permission error: Missing or invalid credentials":**

* LakeXpress auto-configures a `lakexpress_storage` secret from the `--target_storage_id` credential on every publish, so this usually means that credential itself is misconfigured (wrong keys, an unresolvable profile, etc.)
* If querying published views outside of a LakeXpress run, create a secret in MotherDuck manually with your cloud storage credentials (see [Storage Access](/latest/integrations/motherduck-publishing.md#3-storage-access-from-motherduck-automatic-with-a-manual-fallback))
* Verify secrets are configured: `SELECT * FROM duckdb_secrets();`
* Test access: `SELECT count(*) FROM read_parquet('s3://your-bucket/path/*.parquet');`

**"Cannot read file" (external views):**

* Verify cloud storage credentials in MotherDuck
* Check that the S3/GCS/Azure path contains valid Parquet files
* Ensure the bucket allows access from MotherDuck
* Verify the files exist: `aws s3 ls s3://your-bucket/path/ --recursive | head`

**"Connection timeout":**

* Check network connectivity
* MotherDuck requires outbound HTTPS
* Verify no firewall is blocking the connection

### Verifying Setup[​](#verifying-setup "Direct link to Verifying Setup")

Test MotherDuck connectivity:

```
duckdb "md:my_analytics?motherduck_token=your_token" -c "SELECT 1"
```

Test cloud storage access from MotherDuck:

```
SELECT * FROM read_parquet('s3://your-bucket/path/to/file.parquet') LIMIT 1;
```

### Configuring Storage Secrets[​](#configuring-storage-secrets "Direct link to Configuring Storage Secrets")

See [Storage Access from MotherDuck](/latest/integrations/motherduck-publishing.md#3-storage-access-from-motherduck-automatic-with-a-manual-fallback) in Prerequisites for detailed instructions on configuring S3, GCS, or Azure secrets.

## Comparison with Other Targets[​](#comparison-with-other-targets "Direct link to Comparison with Other Targets")

| Feature         | MotherDuck                  | BigQuery              | Snowflake                 | DuckLake                                             |
| --------------- | --------------------------- | --------------------- | ------------------------- | ---------------------------------------------------- |
| Query Engine    | DuckDB                      | BigQuery              | Snowflake                 | DuckDB                                               |
| External Tables | Views with `read_parquet()` | External tables       | External tables           | DuckLake catalog                                     |
| Native Tables   | CREATE TABLE AS SELECT      | Load jobs             | COPY INTO                 | Only for the `upsert` and `delete_insert` strategies |
| Serverless      | Yes                         | Yes                   | No (warehouse required)   | N/A (local)                                          |
| SQL Dialect     | DuckDB/PostgreSQL           | BigQuery SQL          | Snowflake SQL             | DuckDB/PostgreSQL                                    |
| Best For        | Analytics, exploration      | Large-scale analytics | Enterprise data warehouse | Local development                                    |

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/motherduck.svg)MotherDuck Publishing](/latest/integrations/motherduck-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options


---

# Amazon Redshift Publishing

LakeXpress **creates Amazon Redshift tables** from exported Parquet files stored in S3, supporting both native internal tables and Spectrum external tables.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Redshift Cluster[​](#1-redshift-cluster "Direct link to 1. Redshift Cluster")

A Redshift Provisioned cluster or Redshift Serverless workgroup with:

* A database (default: `dev`)
* A user with `CREATE SCHEMA` and `CREATE TABLE` privileges
* Security group allowing inbound TCP on port 5439

### 2. IAM Role[​](#2-iam-role "Direct link to 2. IAM Role")

An IAM role that Redshift can assume, with:

* **S3 read access** on the bucket containing exported Parquet files (required for both internal and external tables)
* **AWS Glue catalog access** (required only for external/Spectrum tables)

Example trust policy:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "redshift.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}
```

### 3. S3 Storage[​](#3-s3-storage "Direct link to 3. S3 Storage")

Both the Redshift cluster and the S3 bucket must be in the **same AWS region** to avoid cross-region transfer costs.

The S3 credential and the Redshift credential must both be in your `credentials.json`:

```
{
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-bucket/lakexpress/",
      "profile": "lakexpress"
    }
  },
  "redshift_classic": {
    "ds_type": "redshift",
    "auth_mode": "classic",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "username": "$env{LX_REDSHIFT_USER}",
      "password": "$env{LX_REDSHIFT_PASSWORD}",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

* Classic (Username/Password)
* IAM

```
{
  "redshift_classic": {
    "ds_type": "redshift",
    "auth_mode": "classic",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "username": "$env{LX_REDSHIFT_USER}",
      "password": "$env{LX_REDSHIFT_PASSWORD}",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

| Field      | Required | Description                                                                                                    |
| ---------- | -------- | -------------------------------------------------------------------------------------------------------------- |
| `host`     | Yes      | Cluster endpoint                                                                                               |
| `port`     | No       | Default: 5439                                                                                                  |
| `database` | No       | Default: `dev`                                                                                                 |
| `username` | Yes      | Database user                                                                                                  |
| `password` | Yes      | Database password                                                                                              |
| `iam_role` | Yes      | IAM role ARN for S3/Glue access. Publishing stops with an error without it                                     |
| `region`   | No       | AWS region (default: `us-east-1`)                                                                              |
| `timeout`  | No       | Connection socket timeout in seconds (default: 600, or the `LAKEXPRESS_REDSHIFT_TIMEOUT` environment variable) |

```
{
  "redshift_iam": {
    "ds_type": "redshift",
    "auth_mode": "iam",
    "info": {
      "host": "my-cluster.eu-west-1.redshift.amazonaws.com",
      "port": 5439,
      "database": "dev",
      "cluster_identifier": "my-cluster",
      "iam_role": "arn:aws:iam::123456789012:role/MyRedshiftRole",
      "region": "eu-west-1"
    }
  }
}
```

| Field                | Required | Description                                                                                                    |
| -------------------- | -------- | -------------------------------------------------------------------------------------------------------------- |
| `host`               | Yes      | Cluster endpoint                                                                                               |
| `port`               | No       | Default: 5439                                                                                                  |
| `database`           | No       | Default: `dev`                                                                                                 |
| `cluster_identifier` | Yes      | Redshift cluster identifier                                                                                    |
| `iam_role`           | Yes      | IAM role ARN. Publishing stops with an error without it                                                        |
| `region`             | No       | AWS region (default: `us-east-1`)                                                                              |
| `timeout`            | No       | Connection socket timeout in seconds (default: 600, or the `LAKEXPRESS_REDSHIFT_TIMEOUT` environment variable) |

## Table Types[​](#table-types "Direct link to Table Types")

### Internal Tables (`--publish_method internal`)[​](#internal-tables---publish_method-internal "Direct link to internal-tables---publish_method-internal")

Creates native Redshift tables and loads data using `COPY FROM S3 FORMAT AS PARQUET`.

* Data is **copied into Redshift storage** -- queries are fast but storage costs apply
* The COPY command reads all Parquet files in the S3 directory (including partitioned exports)
* Decimal columns preserve source precision and scale (e.g., `DECIMAL(12,2)`) -- the same mapping used for external tables; `DECIMAL(38,18)` is used only when the source has no declared precision
* Supports `--pk_constraints` for informational primary key constraints

### External Tables (`--publish_method external`)[​](#external-tables---publish_method-external "Direct link to external-tables---publish_method-external")

Creates Redshift Spectrum tables backed by the AWS Glue Data Catalog.

* Data **stays in S3** -- no data movement, no Redshift storage costs
* Requires an IAM role with Glue catalog permissions
* The Glue database is created automatically (`CREATE EXTERNAL DATABASE IF NOT EXISTS`)
* Decimal columns preserve source precision (e.g., `DECIMAL(12,2)`) -- same mapping used for internal tables
* PK constraints are not supported for external tables

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

| Token         | Description                           |
| ------------- | ------------------------------------- |
| `{schema}`    | Source schema name                    |
| `{table}`     | Source table name                     |
| `{subpath}`   | Value of `--sub_path`                 |
| `{date}`      | Current date (YYYYMMDD)               |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)  |
| `{uuid}`      | UUID4 identifier (consistent per run) |

Set via `--publish_schema_pattern` and `--publish_table_pattern`. Default: `{schema}` for `--publish_method internal`, `EXT_{schema}` for the default `external` method (unless set explicitly); `--publish_table_pattern` defaults to `{table}`.

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for Amazon Redshift are in [Examples & Recipes: Amazon Redshift](/latest/guides/recipes/redshift.md).

## Primary Key Constraints[​](#primary-key-constraints "Direct link to Primary Key Constraints")

When `--pk_constraints` is set, LakeXpress adds `PRIMARY KEY` constraints to internal table DDL based on the source table's primary key columns.

These constraints are **informational only** in Redshift -- they are not enforced but are used by the query optimizer for better execution plans.

PK constraints are not supported for external (Spectrum) tables.

## Notes and Limitations[​](#notes-and-limitations "Direct link to Notes and Limitations")

* **COPY parallelism**: The COPY command automatically distributes file reads across Redshift compute slices. Multi-file exports (partitioned tables) benefit from multi-node clusters.
* **Decimal mapping**: Internal and external (Spectrum) tables use identical decimal mapping -- both preserve the source's precision and scale (e.g. `DECIMAL(12,2)`). `DECIMAL(38,18)` is used only as a fallback when the source type reports no precision (e.g. a bare `NUMBER`).
* **Region**: The Redshift cluster and S3 bucket should be in the same AWS region to avoid cross-region data transfer costs.
* **Glue database**: For external tables, the `CREATE EXTERNAL SCHEMA` DDL automatically creates the backing Glue database if it does not exist.
* **Redshift Serverless**: Both Provisioned and Serverless clusters are supported. The connection uses the same host/port/user/password parameters.
* **Driver**: `redshift_connector` is bundled with LakeXpress: nothing to install.

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. A table using `upsert` or `delete_insert` needs a primary key and is always created as an internal table, whatever `--publish_method` is. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [Compatibility](/latest/reference/compatibility.md)Full list of supported platforms
* [![](/img/logos/aws-s3.png)Storage Configuration: S3](/latest/configuration/storage/s3.md)S3 credential setup
* [Amazon Redshift Reference](/latest/integrations/redshift-reference.md)Data type mappings, CLI arguments, and querying


---

# Amazon Redshift Reference

Reference material for [Amazon Redshift Publishing](/latest/integrations/redshift-publishing.md) -- data type mappings, CLI arguments, and querying.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to Redshift-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to Redshift[​](#postgresql-to-redshift "Direct link to PostgreSQL to Redshift")

| PostgreSQL Type              | Redshift Type      |
| ---------------------------- | ------------------ |
| `SMALLINT`                   | `SMALLINT`         |
| `INTEGER`, `INT4`            | `INTEGER`          |
| `BIGINT`, `INT8`             | `BIGINT`           |
| `NUMERIC(p,s)`               | `DECIMAL(p,s)`     |
| `REAL`, `FLOAT4`             | `REAL`             |
| `DOUBLE PRECISION`, `FLOAT8` | `DOUBLE PRECISION` |
| `VARCHAR(n)`                 | `VARCHAR(n)`       |
| `TEXT`                       | `VARCHAR(MAX)`     |
| `CHAR(n)`                    | `CHAR(n)`          |
| `BOOLEAN`                    | `BOOLEAN`          |
| `DATE`                       | `DATE`             |
| `TIME`                       | `VARCHAR(20)`      |
| `TIMESTAMP`                  | `TIMESTAMP`        |
| `TIMESTAMPTZ`                | `VARCHAR(40)`      |
| `BYTEA`                      | `VARBYTE`          |
| `UUID`                       | `VARCHAR(36)`      |
| `JSON`, `JSONB`, `array []`  | `SUPER`            |

### SQL Server to Redshift[​](#sql-server-to-redshift "Direct link to SQL Server to Redshift")

| SQL Server Type                          | Redshift Type      |
| ---------------------------------------- | ------------------ |
| `TINYINT`, `SMALLINT`                    | `SMALLINT`         |
| `INT`                                    | `INTEGER`          |
| `BIGINT`                                 | `BIGINT`           |
| `DECIMAL(p,s)`                           | `DECIMAL(p,s)`     |
| `MONEY`                                  | `DECIMAL(19,4)`    |
| `SMALLMONEY`                             | `DECIMAL(10,4)`    |
| `FLOAT`                                  | `DOUBLE PRECISION` |
| `REAL`                                   | `REAL`             |
| `BIT`                                    | `BOOLEAN`          |
| `VARCHAR(n)`, `NVARCHAR(n)`, `NCHAR(n)`  | `VARCHAR(n)`       |
| `CHAR(n)`                                | `CHAR(n)`          |
| `TEXT`, `NTEXT`                          | `VARCHAR(MAX)`     |
| `DATE`                                   | `DATE`             |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME` | `TIMESTAMP`        |
| `DATETIMEOFFSET`                         | `VARCHAR(40)`      |
| `VARBINARY`                              | `VARBYTE`          |
| `UNIQUEIDENTIFIER`                       | `VARCHAR(36)`      |

### Oracle to Redshift[​](#oracle-to-redshift "Direct link to Oracle to Redshift")

| Oracle Type                   | Redshift Type      |
| ----------------------------- | ------------------ |
| `NUMBER`                      | `DECIMAL(38,18)`   |
| `NUMBER(p,s)`                 | `DECIMAL(p,s)`     |
| `FLOAT`                       | `DOUBLE PRECISION` |
| `VARCHAR2(n)`, `NVARCHAR2(n)` | `VARCHAR(n)`       |
| `CHAR(n)`                     | `CHAR(n)`          |
| `CLOB`, `NCLOB`               | `VARCHAR(MAX)`     |
| `DATE`                        | `DATE`             |
| `TIMESTAMP`                   | `TIMESTAMP`        |
| `RAW`, `BLOB`                 | `VARBYTE`          |

### Teradata to Redshift[​](#teradata-to-redshift "Direct link to Teradata to Redshift")

| Teradata Type                  | Redshift Type      |
| ------------------------------ | ------------------ |
| `BYTEINT`, `SMALLINT`          | `SMALLINT`         |
| `INTEGER`                      | `INTEGER`          |
| `BIGINT`                       | `BIGINT`           |
| `DECIMAL(p,s)`, `NUMERIC(p,s)` | `DECIMAL(p,s)`     |
| `NUMBER`                       | `DECIMAL(38,18)`   |
| `FLOAT`, `REAL`                | `DOUBLE PRECISION` |
| `VARCHAR(n)`                   | `VARCHAR(n)`       |
| `CHAR(n)`                      | `CHAR(n)`          |
| `CLOB`                         | `VARCHAR(MAX)`     |
| `DATE`                         | `DATE`             |
| `TIME`                         | `VARCHAR(20)`      |
| `TIMESTAMP`                    | `TIMESTAMP`        |
| `BYTE`, `VARBYTE`, `BLOB`      | `VARBYTE`          |

### MySQL to Redshift[​](#mysql-to-redshift "Direct link to MySQL to Redshift")

| MySQL Type                                                          | Redshift Type      |
| ------------------------------------------------------------------- | ------------------ |
| `TINYINT`, `SMALLINT`, `YEAR`                                       | `SMALLINT`         |
| `MEDIUMINT`, `INT`                                                  | `INTEGER`          |
| `BIGINT`                                                            | `BIGINT`           |
| `DECIMAL(p,s)`                                                      | `DECIMAL(p,s)`     |
| `FLOAT`, `DOUBLE`                                                   | `DOUBLE PRECISION` |
| `BIT`                                                               | `BOOLEAN`          |
| `VARCHAR(n)`                                                        | `VARCHAR(n)`       |
| `CHAR(n)`                                                           | `CHAR(n)`          |
| `TINYTEXT`                                                          | `VARCHAR(255)`     |
| `TEXT`, `MEDIUMTEXT`, `LONGTEXT`                                    | `VARCHAR(MAX)`     |
| `ENUM`, `SET`                                                       | `VARCHAR(256)`     |
| `DATE`                                                              | `DATE`             |
| `TIME`                                                              | `VARCHAR(20)`      |
| `DATETIME`, `TIMESTAMP`                                             | `TIMESTAMP`        |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `VARBYTE`          |
| `JSON`                                                              | `SUPER`            |

### MariaDB to Redshift[​](#mariadb-to-redshift "Direct link to MariaDB to Redshift")

| MariaDB Type                                                        | Redshift Type      |
| ------------------------------------------------------------------- | ------------------ |
| `TINYINT`, `SMALLINT`, `YEAR`                                       | `SMALLINT`         |
| `MEDIUMINT`, `INT`                                                  | `INTEGER`          |
| `BIGINT`                                                            | `BIGINT`           |
| `DECIMAL(p,s)`                                                      | `DECIMAL(p,s)`     |
| `FLOAT`, `DOUBLE`                                                   | `DOUBLE PRECISION` |
| `BIT`                                                               | `BOOLEAN`          |
| `VARCHAR(n)`                                                        | `VARCHAR(n)`       |
| `CHAR(n)`                                                           | `CHAR(n)`          |
| `TINYTEXT`                                                          | `VARCHAR(255)`     |
| `TEXT`, `MEDIUMTEXT`, `LONGTEXT`                                    | `VARCHAR(MAX)`     |
| `ENUM`, `SET`                                                       | `VARCHAR(256)`     |
| `DATE`                                                              | `DATE`             |
| `TIME`                                                              | `VARCHAR(20)`      |
| `DATETIME`, `TIMESTAMP`                                             | `TIMESTAMP`        |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `VARBYTE`          |
| `JSON`                                                              | `SUPER`            |

### SAP HANA to Redshift[​](#sap-hana-to-redshift "Direct link to SAP HANA to Redshift")

| SAP HANA Type                                         | Redshift Type      |
| ----------------------------------------------------- | ------------------ |
| `TINYINT`, `SMALLINT`                                 | `SMALLINT`         |
| `INTEGER`                                             | `INTEGER`          |
| `BIGINT`                                              | `BIGINT`           |
| `DECIMAL(p,s)`                                        | `DECIMAL(p,s)`     |
| `SMALLDECIMAL`, `CLOB`, `NCLOB`, `TEXT`, `SECONDDATE` | `VARCHAR(MAX)`     |
| `REAL`                                                | `REAL`             |
| `DOUBLE`                                              | `DOUBLE PRECISION` |
| `BOOLEAN`                                             | `BOOLEAN`          |
| `VARCHAR(n)`, `NVARCHAR(n)`, `NCHAR(n)`               | `VARCHAR(n)`       |
| `CHAR(n)`                                             | `CHAR(n)`          |
| `DATE`                                                | `DATE`             |
| `TIME`                                                | `VARCHAR(20)`      |
| `TIMESTAMP`                                           | `TIMESTAMP`        |
| `VARBINARY`, `BLOB`                                   | `VARBYTE`          |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Redshift Publishing Arguments[​](#redshift-publishing-arguments "Direct link to Redshift Publishing Arguments")

| Option                                | Type    | Description                                                                                                                         |
| ------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `--publish_target ID`                 | String  | Credential ID for Redshift publishing (required)                                                                                    |
| `--publish_method internal\|external` | String  | Table type: native Redshift table (`internal`) or Spectrum external table (`external`)                                              |
| `--publish_schema_pattern PATTERN`    | String  | Schema naming pattern (default: `{schema}` for `internal`, `EXT_{schema}` for the default `external` method, unless set explicitly) |
| `--publish_table_pattern PATTERN`     | String  | Table naming pattern (default: `{table}`)                                                                                           |
| `--sub_path PATH`                     | String  | Value substituted for the `{subpath}` naming token                                                                                  |
| `--pk_constraints`                    | Flag    | Add informational `PRIMARY KEY` constraints (internal tables only)                                                                  |
| `--n_jobs N`                          | Integer | Parallel workers for table creation (default: 1)                                                                                    |

## Querying Redshift Tables[​](#querying-redshift-tables "Direct link to Querying Redshift Tables")

**Internal table:**

```
SELECT * FROM int_mydata_20260730.customer LIMIT 10;
```

**External table (Spectrum):**

```
SELECT * FROM ext_mydata_20260730.customer LIMIT 10;
```

## See Also[​](#see-also "Direct link to See Also")

* [Amazon Redshift Publishing](/latest/integrations/redshift-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options
* [![](/img/logos/aws-s3.png)Storage Configuration: S3](/latest/configuration/storage/s3.md)S3 credential setup


---

# Snowflake Publishing

LakeXpress **creates Snowflake tables** from exported Parquet files, enabling queries on data stored in S3, Azure, or GCS.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### 1. Snowflake Setup[​](#1-snowflake-setup "Direct link to 1. Snowflake Setup")

LakeXpress needs a **warehouse**, a **database** and an **external stage** on the folder where it writes the Parquet files. It creates everything else itself: the target schemas (`CREATE SCHEMA IF NOT EXISTS`), the tables (`CREATE TABLE` / `CREATE EXTERNAL TABLE`), the views of external tables, and the `COPY INTO` of internal tables. The Parquet file format is given inline, so no `FILE FORMAT` object is needed.

The scripts below create a dedicated role and user with only the privileges LakeXpress needs. The object names are examples: keep them consistent with your `credentials.json`.

#### Step 1: Warehouse, database, role and user[​](#step-1-warehouse-database-role-and-user "Direct link to Step 1: Warehouse, database, role and user")

```
-- Warehouse and database
USE ROLE SYSADMIN;
CREATE WAREHOUSE IF NOT EXISTS LAKEXPRESS_WH
  WAREHOUSE_SIZE = 'XSMALL'
  AUTO_SUSPEND = 60
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE;
CREATE DATABASE IF NOT EXISTS LAKEXPRESS_DB;

-- Role and user for LakeXpress
USE ROLE SECURITYADMIN;
CREATE ROLE IF NOT EXISTS LAKEXPRESS_ROLE;
GRANT ROLE LAKEXPRESS_ROLE TO ROLE SYSADMIN;
CREATE USER IF NOT EXISTS LAKEXPRESS_USER
  PASSWORD = '<strong-password>'
  DEFAULT_ROLE = LAKEXPRESS_ROLE
  DEFAULT_WAREHOUSE = LAKEXPRESS_WH;
GRANT ROLE LAKEXPRESS_ROLE TO USER LAKEXPRESS_USER;

-- Privileges (granted by SYSADMIN, owner of the warehouse and the database)
USE ROLE SYSADMIN;
GRANT USAGE ON WAREHOUSE LAKEXPRESS_WH TO ROLE LAKEXPRESS_ROLE;
GRANT USAGE, CREATE SCHEMA ON DATABASE LAKEXPRESS_DB TO ROLE LAKEXPRESS_ROLE;
GRANT USAGE ON SCHEMA LAKEXPRESS_DB.PUBLIC TO ROLE LAKEXPRESS_ROLE;  -- schema of the stage
```

LakeXpress owns the schemas it creates, so it needs no other privilege on them. If it publishes into a schema that already exists and that another role owns, also grant:

```
GRANT USAGE, CREATE TABLE, CREATE EXTERNAL TABLE, CREATE VIEW
  ON SCHEMA LAKEXPRESS_DB.<SCHEMA> TO ROLE LAKEXPRESS_ROLE;
```

For key pair or PAT authentication instead of a password, see [Authentication Modes](#authentication-modes).

#### Step 2: Storage integration and stage[​](#step-2-storage-integration-and-stage "Direct link to Step 2: Storage integration and stage")

The stage must point to **exactly the folder** of the storage entry LakeXpress exports to (its `directory`): LakeXpress reads each table under `<stage>/<sub_path>/<source schema>/<table>/`. A storage integration lets Snowflake read that folder without storing cloud keys in Snowflake. Snowflake only reads the files, so read access is enough.

* AWS S3
* Azure Storage
* Google Cloud Storage

In AWS, create an IAM policy that can read the export folder, and an IAM role (for example `snowflake-lakexpress`) with that policy:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::my-datalake-bucket/lakexpress/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::my-datalake-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["lakexpress/*"] } }
    }
  ]
}
```

Then, in Snowflake:

```
USE ROLE ACCOUNTADMIN;
CREATE STORAGE INTEGRATION IF NOT EXISTS LAKEXPRESS_S3_INT
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-lakexpress'
  STORAGE_ALLOWED_LOCATIONS = ('s3://my-datalake-bucket/lakexpress/');

-- Note STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID
DESC INTEGRATION LAKEXPRESS_S3_INT;

GRANT USAGE ON INTEGRATION LAKEXPRESS_S3_INT TO ROLE SYSADMIN;

USE ROLE SYSADMIN;
CREATE STAGE IF NOT EXISTS LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE
  URL = 's3://my-datalake-bucket/lakexpress/'
  STORAGE_INTEGRATION = LAKEXPRESS_S3_INT;
GRANT USAGE ON STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE TO ROLE LAKEXPRESS_ROLE;
```

Back in AWS, set the trust policy of the IAM role with the two values returned by `DESC INTEGRATION`:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<STORAGE_AWS_IAM_USER_ARN>" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<STORAGE_AWS_EXTERNAL_ID>" } }
    }
  ]
}
```

The matching storage entry has `"directory": "s3://my-datalake-bucket/lakexpress/"`.

```
USE ROLE ACCOUNTADMIN;
CREATE STORAGE INTEGRATION IF NOT EXISTS LAKEXPRESS_AZURE_INT
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'AZURE'
  ENABLED = TRUE
  AZURE_TENANT_ID = '<tenant-id>'
  STORAGE_ALLOWED_LOCATIONS = ('azure://<account>.blob.core.windows.net/<container>/lakexpress/');

-- Open AZURE_CONSENT_URL to approve the Snowflake application, and note AZURE_MULTI_TENANT_APP_NAME
DESC INTEGRATION LAKEXPRESS_AZURE_INT;

GRANT USAGE ON INTEGRATION LAKEXPRESS_AZURE_INT TO ROLE SYSADMIN;

USE ROLE SYSADMIN;
CREATE STAGE IF NOT EXISTS LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE
  URL = 'azure://<account>.blob.core.windows.net/<container>/lakexpress/'
  STORAGE_INTEGRATION = LAKEXPRESS_AZURE_INT;
GRANT USAGE ON STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE TO ROLE LAKEXPRESS_ROLE;
```

In the Azure portal, give the Snowflake application (`AZURE_MULTI_TENANT_APP_NAME`) the **Storage Blob Data Reader** role on the container. The stage uses Snowflake's `azure://` form of the folder that the LakeXpress storage entry names `abfss://<container>@<account>.dfs.core.windows.net/lakexpress/`.

```
USE ROLE ACCOUNTADMIN;
CREATE STORAGE INTEGRATION IF NOT EXISTS LAKEXPRESS_GCS_INT
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'GCS'
  ENABLED = TRUE
  STORAGE_ALLOWED_LOCATIONS = ('gcs://my-datalake-bucket/lakexpress/');

-- Note STORAGE_GCP_SERVICE_ACCOUNT
DESC INTEGRATION LAKEXPRESS_GCS_INT;

GRANT USAGE ON INTEGRATION LAKEXPRESS_GCS_INT TO ROLE SYSADMIN;

USE ROLE SYSADMIN;
CREATE STAGE IF NOT EXISTS LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE
  URL = 'gcs://my-datalake-bucket/lakexpress/'
  STORAGE_INTEGRATION = LAKEXPRESS_GCS_INT;
GRANT USAGE ON STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE TO ROLE LAKEXPRESS_ROLE;
```

In Google Cloud, give the service account `STORAGE_GCP_SERVICE_ACCOUNT` the **Storage Object Viewer** role on the bucket. The stage uses Snowflake's `gcs://` form of the folder that the LakeXpress storage entry names `gs://my-datalake-bucket/lakexpress/`.

Before publishing, LakeXpress checks that the stage URL is the storage `directory`, and stops with an error naming both if they differ. Snowflake's `gcs://` and `azure://` forms count as the same folder as LakeXpress's `gs://` and `abfss://` ones.

The stage can live in another schema: give its full name in `credentials.json` (`"stage": "LAKEXPRESS_DB.LANDING.LAKEXPRESS_STAGE"`) and grant `USAGE` on that schema instead of `PUBLIC`. A stage name without a schema is looked up in `PUBLIC`.

#### Step 3: Check the setup as the LakeXpress user[​](#step-3-check-the-setup-as-the-lakexpress-user "Direct link to Step 3: Check the setup as the LakeXpress user")

```
USE ROLE LAKEXPRESS_ROLE;
USE WAREHOUSE LAKEXPRESS_WH;
DESC STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE;  -- the URL must be the storage directory
LIST @LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE;       -- after a first export: the Parquet files
```

### 2. Cloud Storage Setup[​](#2-cloud-storage-setup "Direct link to 2. Cloud Storage Setup")

Both S3 and Snowflake credentials must be in your `credentials.json`:

```
{
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://my-datalake-bucket/lakexpress/",
      "profile": "lakexpress"
    }
  },
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "password",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "password": "$env{LX_SNOWFLAKE_PASSWORD}",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE",
      "role": "LAKEXPRESS_ROLE"
    }
  }
}
```

`role` is optional: without it, LakeXpress uses the user's default role. The connection also accepts an optional `schema`.

**Critical:** The storage `directory` path **must exactly match** the Snowflake stage location.

**Verify the match:**

```
DESC STAGE LAKEXPRESS_DB.PUBLIC.LAKEXPRESS_STAGE;
-- URL column shows: s3://my-datalake-bucket/lakexpress/

-- This MUST match your S3 credentials directory
-- "directory": "s3://my-datalake-bucket/lakexpress/"  ✓ Correct
-- "directory": "s3://my-datalake-bucket/"              ✗ Wrong - files won't be found
```

## Authentication Modes[​](#authentication-modes "Direct link to Authentication Modes")

* Password
* Programmatic Access Token (PAT)

```
{
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "password",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "password": "$env{LX_SNOWFLAKE_PASSWORD}",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE"
    }
  }
}
```

Recommended for automation:

```
{
  "snowflake_prod": {
    "ds_type": "snowflake",
    "auth_mode": "pat",
    "info": {
      "account": "myorg-myaccount",
      "user": "LAKEXPRESS_USER",
      "token": "your-personal-access-token",
      "warehouse": "LAKEXPRESS_WH",
      "database": "LAKEXPRESS_DB",
      "stage": "LAKEXPRESS_STAGE"
    }
  }
}
```

PATs require a network policy configured for the user.

**PAT benefits:**

* Easily rotated, configurable lifetime
* No MFA prompts -- suited for CI/CD pipelines

### Other Authentication Modes[​](#other-authentication-modes "Direct link to Other Authentication Modes")

LakeXpress also supports two additional `auth_mode` values not shown above:

* `keypair` -- RSA private/public key pair authentication (`info.private_key_path`, optional `info.private_key_passphrase`)
* `oauth` -- OAuth token authentication (`info.oauth_token` or `info.authenticator`)

## Table Types[​](#table-types "Direct link to Table Types")

### External Tables[​](#external-tables "Direct link to External Tables")

Data stays in cloud storage; Snowflake stores only metadata. Views are auto-created for cleaner querying.

* No data loading time or Snowflake storage costs
* Immediate availability after export
* Suited for: data lake integration, infrequent access, exploration, ETL staging

Options for `config create`:

```
--publish_target snowflake_prod
--publish_method external  # Default
```

### Internal Tables[​](#internal-tables "Direct link to Internal Tables")

Data is loaded into Snowflake storage with full feature support (clustering, time travel, etc.). Can include PRIMARY KEY constraints from source.

* Faster queries with full Snowflake features
* Maintains PRIMARY KEY constraints
* Suited for: frequently queried data, production warehouses, complex joins

Options for `config create`:

```
--publish_target snowflake_prod
--publish_method internal
--pk_constraints  # Optional: include PK constraints
```

## Dynamic Naming Patterns[​](#dynamic-naming-patterns "Direct link to Dynamic Naming Patterns")

Schema and table names support token-based patterns for time-partitioned exports, schema consolidation, or unique identifiers.

### Supported Tokens[​](#supported-tokens "Direct link to Supported Tokens")

| Token         | Description                                                     | Example Output    |
| ------------- | --------------------------------------------------------------- | ----------------- |
| `{schema}`    | Source schema name                                              | `tpch_1`          |
| `{table}`     | Source table name                                               | `customer`        |
| `{database}`  | Target Snowflake database (`database` of the credentials entry) | `ANALYTICS`       |
| `{date}`      | Current date (YYYYMMDD)                                         | `20251120`        |
| `{timestamp}` | Current timestamp (YYYYMMDD\_HHMMSS)                            | `20251120_112511` |
| `{uuid}`      | UUID4 identifier (consistent per run)                           | `a1b2c3d4-...`    |
| `{subpath}`   | CLI `--sub_path` parameter                                      | `toto`            |

**Note:** All timestamps are in local time. The `{table}` token is **mandatory** for `--publish_table_pattern`.

### CLI Arguments[​](#cli-arguments "Direct link to CLI Arguments")

```
--publish_schema_pattern PATTERN    # Schema name pattern (default varies by table type)
--publish_table_pattern PATTERN      # Table name pattern (must include {table} token)
```

### Default Patterns[​](#default-patterns "Direct link to Default Patterns")

| Table Type | Default Schema Pattern | Default Table Pattern |
| ---------- | ---------------------- | --------------------- |
| External   | `EXT_{schema}`         | `{table}`             |
| Internal   | `{schema}`             | `{table}`             |

### Common Patterns[​](#common-patterns "Direct link to Common Patterns")

Time-Based Partitioning

```
--publish_schema_pattern "EXT_{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target snowflake_prod

# Results:
# Schema: EXT_tpch_1_20251120
# Tables: customer, orders, lineitem
```

Multi-Schema Consolidation

```
--source_schema_name schema1,schema2,schema3
--publish_schema_pattern "{subpath}"
--publish_table_pattern "{schema}_{table}"
--sub_path consolidated
--publish_target snowflake_prod

# Results:
# Schema: consolidated
# Tables: schema1_customer, schema2_customer, schema3_customer
```

Unique Run Identifiers

```
--publish_schema_pattern "EXT_{schema}_{uuid}"
--publish_table_pattern "{table}"
--publish_target snowflake_prod

# Results:
# Schema: EXT_tpch_1_a1b2c3d4-e5f6-7890-abcd-ef1234567890
# Tables: customer, orders, lineitem
```

## Usage Examples[​](#usage-examples "Direct link to Usage Examples")

Ready-to-run commands for Snowflake are in [Examples & Recipes: Snowflake](/latest/guides/recipes/snowflake.md).

## Views and Customization[​](#views-and-customization "Direct link to Views and Customization")

### Automatic Views (External Tables Only)[​](#automatic-views-external-tables-only "Direct link to Automatic Views (External Tables Only)")

For external tables, LakeXpress also creates a `V_<table>` view. The external table exposes the typed columns plus the raw Parquet `VALUE` column; the view leaves `VALUE` out.

LakeXpress creates every identifier quoted, so names keep the case of the source. With a lowercase source schema `tpch_1`, quote them in queries:

```
-- External table: typed columns + VALUE
SELECT * FROM "EXT_tpch_1"."customer";

-- View: typed columns only
SELECT * FROM "EXT_tpch_1"."V_customer";
```

### View Options[​](#view-options "Direct link to View Options")

| Option       | Description        | Default       |
| ------------ | ------------------ | ------------- |
| `--no_views` | Skip view creation | Views created |

The view name prefix is **not configurable** -- it is hardcoded to `V_` (there is no `--snowflake_view_prefix` CLI flag).

**No views:**

Options for `config create`:

```
--publish_target snowflake_prod
--publish_method external
--no_views

# Tables: customer, orders
# Views: None
```

### Refreshing External Tables[​](#refreshing-external-tables "Direct link to Refreshing External Tables")

Each publish drops and recreates the external tables with `REFRESH_ON_CREATE = TRUE` (and `AUTO_REFRESH = FALSE`), so they list the files present at publish time. Refresh by hand only if files change outside a LakeXpress run:

```
ALTER EXTERNAL TABLE "EXT_tpch_1"."customer" REFRESH;
```

Or via Snowflake CLI:

```
snow sql -q 'ALTER EXTERNAL TABLE "EXT_tpch_1"."customer" REFRESH;' -c lakexpress
```

## Primary Key Constraints[​](#primary-key-constraints "Direct link to Primary Key Constraints")

Applies to **internal tables** only. LakeXpress propagates PRIMARY KEY constraints from the source database to Snowflake.

### Usage[​](#usage "Direct link to Usage")

```
--publish_target snowflake_prod
--publish_method internal
--pk_constraints
```

### How It Works[​](#how-it-works "Direct link to How It Works")

1. Reads PRIMARY KEY metadata from the source database
2. Stores PK information in the LakeXpress DB
3. Adds PRIMARY KEY constraints to the Snowflake DDL
4. Snowflake stores the constraints as metadata. It does not enforce them (duplicates are not rejected), but they document the model for BI and modeling tools

### Requirements[​](#requirements "Direct link to Requirements")

* `--publish_method internal` only
* Source database must have PRIMARY KEY constraints defined
* LakeXpress DB must have `is_primary_key` column (auto-created in v0.1.16+)

### Benefits[​](#benefits "Direct link to Benefits")

* Documents the data model in Snowflake
* BI and data modeling tools read the keys to infer relationships

LakeXpress creates plain `PRIMARY KEY` constraints, without `RELY`, so the Snowflake optimizer does not use them.

## Write Strategies[​](#write-strategies "Direct link to Write Strategies")

Incremental tables support the `append` (default), `upsert` and `delete_insert` strategies. A table using `upsert` or `delete_insert` needs a primary key and is always created as an internal table, whatever `--publish_method` is. See [Loading Strategies](/latest/advanced/incremental/loading-strategies.md).

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/snowflake.svg)Snowflake Reference](/latest/integrations/snowflake-reference.md)Data type mappings, CLI arguments, deprecated options
* [CLI Reference](/latest/cli/.md)All command-line options
* [Examples & Recipes](/latest/guides/command-cookbook.md)Working command examples


---

# Snowflake Reference

Reference material for [Snowflake Publishing](/latest/integrations/snowflake-publishing.md) -- data type mappings and CLI arguments.

## Data Type Mapping[​](#data-type-mapping "Direct link to Data Type Mapping")

Source types map automatically to Snowflake-compatible types. LakeXpress exports through Parquet, so the mapping is driven by the normalized source type name.

Pick your source database:

* PostgreSQL
* SQL Server
* Oracle
* Teradata
* MySQL
* MariaDB
* SAP HANA

### PostgreSQL to Snowflake[​](#postgresql-to-snowflake "Direct link to PostgreSQL to Snowflake")

| PostgreSQL Type                                 | Snowflake Type  |
| ----------------------------------------------- | --------------- |
| `SMALLINT`, `INTEGER`, `INT4`, `BIGINT`, `INT8` | `NUMBER`        |
| `NUMERIC(p,s)`                                  | `NUMBER(p,s)`   |
| `REAL`, `FLOAT4`, `DOUBLE PRECISION`, `FLOAT8`  | `FLOAT`         |
| `VARCHAR(n)`, `CHAR(n)`                         | `VARCHAR(n)`    |
| `TEXT`                                          | `VARCHAR`       |
| `BOOLEAN`                                       | `BOOLEAN`       |
| `DATE`                                          | `DATE`          |
| `TIME`                                          | `TIME`          |
| `TIMESTAMP`                                     | `TIMESTAMP_NTZ` |
| `TIMESTAMPTZ`                                   | `TIMESTAMP_TZ`  |
| `BYTEA`                                         | `BINARY`        |
| `UUID`                                          | `VARCHAR(36)`   |
| `JSON`, `JSONB`, `array []`                     | `VARIANT`       |

### SQL Server to Snowflake[​](#sql-server-to-snowflake "Direct link to SQL Server to Snowflake")

| SQL Server Type                                    | Snowflake Type  |
| -------------------------------------------------- | --------------- |
| `TINYINT`, `SMALLINT`, `INT`, `BIGINT`             | `NUMBER`        |
| `DECIMAL(p,s)`                                     | `NUMBER(p,s)`   |
| `MONEY`                                            | `NUMBER(19,4)`  |
| `SMALLMONEY`                                       | `NUMBER(10,4)`  |
| `FLOAT`, `REAL`                                    | `FLOAT`         |
| `BIT`                                              | `BOOLEAN`       |
| `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)` | `VARCHAR(n)`    |
| `TEXT`, `NTEXT`                                    | `VARCHAR`       |
| `DATE`                                             | `DATE`          |
| `DATETIME`, `DATETIME2`, `SMALLDATETIME`           | `TIMESTAMP_NTZ` |
| `DATETIMEOFFSET`                                   | `TIMESTAMP_TZ`  |
| `VARBINARY`                                        | `BINARY`        |
| `UNIQUEIDENTIFIER`                                 | `VARCHAR(36)`   |

### Oracle to Snowflake[​](#oracle-to-snowflake "Direct link to Oracle to Snowflake")

| Oracle Type                              | Snowflake Type  |
| ---------------------------------------- | --------------- |
| `NUMBER`                                 | `NUMBER`        |
| `NUMBER(p,s)`                            | `NUMBER(p,s)`   |
| `FLOAT`                                  | `FLOAT`         |
| `VARCHAR2(n)`, `NVARCHAR2(n)`, `CHAR(n)` | `VARCHAR(n)`    |
| `CLOB`, `NCLOB`                          | `VARCHAR`       |
| `DATE`                                   | `DATE`          |
| `TIMESTAMP`                              | `TIMESTAMP_NTZ` |
| `RAW`, `BLOB`                            | `BINARY`        |

### Teradata to Snowflake[​](#teradata-to-snowflake "Direct link to Teradata to Snowflake")

| Teradata Type                             | Snowflake Type  |
| ----------------------------------------- | --------------- |
| `BYTEINT`, `CLOB`, `VARBYTE`              | `VARCHAR`       |
| `SMALLINT`, `INTEGER`, `BIGINT`, `NUMBER` | `NUMBER`        |
| `DECIMAL(p,s)`, `NUMERIC(p,s)`            | `NUMBER(p,s)`   |
| `FLOAT`, `REAL`                           | `FLOAT`         |
| `VARCHAR(n)`, `CHAR(n)`                   | `VARCHAR(n)`    |
| `DATE`                                    | `DATE`          |
| `TIME`                                    | `TIME`          |
| `TIMESTAMP`                               | `TIMESTAMP_NTZ` |
| `BYTE`, `BLOB`                            | `BINARY`        |

### MySQL to Snowflake[​](#mysql-to-snowflake "Direct link to MySQL to Snowflake")

| MySQL Type                                                          | Snowflake Type  |
| ------------------------------------------------------------------- | --------------- |
| `TINYINT`, `SMALLINT`, `MEDIUMINT`, `INT`, `BIGINT`, `YEAR`         | `NUMBER`        |
| `DECIMAL(p,s)`                                                      | `NUMBER(p,s)`   |
| `FLOAT`, `DOUBLE`                                                   | `FLOAT`         |
| `BIT`                                                               | `BOOLEAN`       |
| `VARCHAR(n)`, `CHAR(n)`                                             | `VARCHAR(n)`    |
| `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`         | `VARCHAR`       |
| `DATE`                                                              | `DATE`          |
| `TIME`                                                              | `TIME`          |
| `DATETIME`, `TIMESTAMP`                                             | `TIMESTAMP_NTZ` |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `BINARY`        |
| `JSON`                                                              | `VARIANT`       |

### MariaDB to Snowflake[​](#mariadb-to-snowflake "Direct link to MariaDB to Snowflake")

| MariaDB Type                                                        | Snowflake Type  |
| ------------------------------------------------------------------- | --------------- |
| `TINYINT`, `SMALLINT`, `MEDIUMINT`, `INT`, `BIGINT`, `YEAR`         | `NUMBER`        |
| `DECIMAL(p,s)`                                                      | `NUMBER(p,s)`   |
| `FLOAT`, `DOUBLE`                                                   | `FLOAT`         |
| `BIT`                                                               | `BOOLEAN`       |
| `VARCHAR(n)`, `CHAR(n)`                                             | `VARCHAR(n)`    |
| `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT`, `ENUM`, `SET`         | `VARCHAR`       |
| `DATE`                                                              | `DATE`          |
| `TIME`                                                              | `TIME`          |
| `DATETIME`, `TIMESTAMP`                                             | `TIMESTAMP_NTZ` |
| `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | `BINARY`        |
| `JSON`                                                              | `VARIANT`       |

### SAP HANA to Snowflake[​](#sap-hana-to-snowflake "Direct link to SAP HANA to Snowflake")

| SAP HANA Type                                         | Snowflake Type  |
| ----------------------------------------------------- | --------------- |
| `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`            | `NUMBER`        |
| `DECIMAL(p,s)`                                        | `NUMBER(p,s)`   |
| `SMALLDECIMAL`, `CLOB`, `NCLOB`, `TEXT`, `SECONDDATE` | `VARCHAR`       |
| `REAL`, `DOUBLE`                                      | `FLOAT`         |
| `BOOLEAN`                                             | `BOOLEAN`       |
| `VARCHAR(n)`, `NVARCHAR(n)`, `CHAR(n)`, `NCHAR(n)`    | `VARCHAR(n)`    |
| `DATE`                                                | `DATE`          |
| `TIME`                                                | `TIME`          |
| `TIMESTAMP`                                           | `TIMESTAMP_NTZ` |
| `VARBINARY`, `BLOB`                                   | `BINARY`        |

## CLI Reference[​](#cli-reference "Direct link to CLI Reference")

### Snowflake Publishing Arguments[​](#snowflake-publishing-arguments "Direct link to Snowflake Publishing Arguments")

| Option                                 | Type   | Description                                                                           |
| -------------------------------------- | ------ | ------------------------------------------------------------------------------------- |
| `--publish_target ID`                  | String | Credential ID for Snowflake publishing                                                |
| `--publish_schema_pattern PATTERN`     | String | Schema naming pattern (default: `EXT_{schema}` for external, `{schema}` for internal) |
| `--publish_table_pattern PATTERN`      | String | Table naming pattern (default: `{table}`)                                             |
| `--publish_method {external,internal}` | Enum   | Table type to publish (default: `external`)                                           |
| `--no_views`                           | Flag   | Skip view creation (external tables only)                                             |
| `--pk_constraints`                     | Flag   | Add PRIMARY KEY constraints (internal tables only)                                    |

The view name prefix is hardcoded to `V_` -- there is no CLI flag to change it.

### Deprecated / Renamed Arguments[​](#deprecated--renamed-arguments "Direct link to Deprecated / Renamed Arguments")

| Old Argument                 | Current Argument                        | Notes                                                    |
| ---------------------------- | --------------------------------------- | -------------------------------------------------------- |
| `--publish`                  | `--publish_target`                      | Removed in v0.1.12+                                      |
| `--snowflake_table_type`     | `--publish_method {external\|internal}` | Renamed to the generic, cross-platform flag              |
| `--snowflake_pk_constraints` | `--pk_constraints`                      | Renamed to the generic, cross-platform flag              |
| `--snowflake_no_views`       | `--no_views`                            | Renamed to the generic, cross-platform flag              |
| `--snowflake_view_prefix`    | *(none)*                                | No longer configurable; view prefix is hardcoded to `V_` |

## See Also[​](#see-also "Direct link to See Also")

* [![](/img/logos/snowflake.svg)Snowflake Publishing](/latest/integrations/snowflake-publishing.md)Setup and usage guide
* [CLI Reference](/latest/cli/.md)All command-line options


---

# Set up the LakeXpress DB

LakeXpress needs a database of its own, the **LakeXpress DB**. It stores the sync configurations created by `config create`, the history and status of every run, and the watermarks of incremental syncs. Every LakeXpress command points to it with `--lxdb_auth_id`, so it must be set up before the first export.

It is separate from the databases you export from: LakeXpress only writes its own bookkeeping tables there.

## Two steps[​](#two-steps "Direct link to Two steps")

1. **You create the database.** On PostgreSQL, SQL Server or MySQL, create an empty database and a user that owns it. SQLite and DuckDB need nothing: the file is created on first use.
2. **LakeXpress creates its tables.** Run `lxdb init` once. It creates the tables and checks that the connection works. The first `config create` or `sync` also creates them if needed.

## Choose a backend[​](#choose-a-backend "Direct link to Choose a backend")

Server database

For production: several users or machines share the same configurations

* [![](/img/logos/postgresql.svg)PostgreSQL](/latest/lakexpress-db/postgresql.md)**Recommended**
* [![](/img/logos/sqlserver.svg)SQL Server](/latest/lakexpress-db/sql-server.md)when SQL Server is your standard database
* [![](/img/logos/mysql.svg)MySQL](/latest/lakexpress-db/mysql.md)when MySQL is your standard database

Local file

For trying LakeXpress and single-user setups. The configurations and run history live in a file on one machine: syncs run from that machine, one at a time

* [SQLite](/latest/lakexpress-db/sqlite.md)
* [![](/img/logos/duckdb.svg)DuckDB](/latest/lakexpress-db/duckdb.md)

## Initialize it[​](#initialize-it "Direct link to Initialize it")

After adding the LakeXpress DB entry to your credentials file (see the page of your backend), create the tables:

* Windows (PowerShell)
* Linux

```
.\LakeXpress.exe lxdb init `
  -a credentials.json `
  --lxdb_auth_id lxdb_postgres
```

```
./LakeXpress lxdb init \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres
```

`lxdb_postgres` is the name of the entry in `credentials.json`. The command can be run again safely: existing tables and data are kept.

To empty or remove the LakeXpress tables later, see `lxdb truncate` and `lxdb drop` in the [CLI Reference](/latest/cli/lxdb.md).


---

# DuckDB as LakeXpress DB

Like SQLite, DuckDB needs no server: the LakeXpress DB is a single file. DuckDB can only be used as the LakeXpress DB, not as a source database.

## 1. Add it to the credentials file[​](#1-add-it-to-the-credentials-file "Direct link to 1. Add it to the credentials file")

There is nothing to create beforehand: LakeXpress creates the file, and any missing folder in its path, on first use.

```
{
  "lxdb_duckdb": {
    "ds_type": "duckdb",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/var/lib/lakexpress/lakexpress_log.duckdb"
    }
  }
}
```

| Parameter   | Required | Description                                                                                                       |
| ----------- | -------- | ----------------------------------------------------------------------------------------------------------------- |
| `ds_type`   | Yes      | Must be `"duckdb"`                                                                                                |
| `auth_mode` | Yes      | Must be `"filesystem"`                                                                                            |
| `filepath`  | Yes      | Path to the DuckDB file. In-memory mode (`:memory:`) is not supported: the run history must persist between runs. |

Use an absolute path. A relative path is resolved against the directory LakeXpress is run from, so running it from another directory would create a new, empty LakeXpress DB there.

## 2. Create the LakeXpress tables[​](#2-create-the-lakexpress-tables "Direct link to 2. Create the LakeXpress tables")

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_duckdb
```

## When to use DuckDB[​](#when-to-use-duckdb "Direct link to When to use DuckDB")

**Good for:** trying LakeXpress, development, single-user setups without a database server.

**Avoid for:** several users or machines sharing configurations, syncs running in parallel.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Unable to open database file:** The user running LakeXpress cannot write to the file or to its folder. Check the permissions of the path in `filepath`, and the free disk space.

**Database is locked:** Another process has the file open, for example another LakeXpress run or a DuckDB client. Close it and run the command again.


---

# MySQL as LakeXpress DB

## 1. Create the database[​](#1-create-the-database "Direct link to 1. Create the database")

Run as a MySQL administrator (for example `root`):

```
CREATE DATABASE lakexpress_log CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'lakexpress_admin'@'%' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON lakexpress_log.* TO 'lakexpress_admin'@'%';
```

`'%'` allows connections from any host. Replace it with the host LakeXpress runs on to restrict access.

## 2. Add it to the credentials file[​](#2-add-it-to-the-credentials-file "Direct link to 2. Add it to the credentials file")

```
{
  "lxdb_mysql": {
    "ds_type": "mysql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "mysql.company.com",
      "port": 3306,
      "database": "lakexpress_log"
    }
  }
}
```

See [MySQL connection parameters](/latest/configuration/databases/mysql.md#connection-parameters) for the full list.

## 3. Create the LakeXpress tables[​](#3-create-the-lakexpress-tables "Direct link to 3. Create the LakeXpress tables")

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_mysql
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**`Access denied for user 'lakexpress_admin'@'%' to database 'lakexpress_log'` (error 1044):** The database does not exist, `database` in the credentials file has a different name, or the `GRANT` was not run. Check with `SHOW DATABASES;` and `SHOW GRANTS FOR 'lakexpress_admin'@'%';`.

**Connection refused or other connection errors:** See the [MySQL troubleshooting](/latest/configuration/databases/mysql.md#troubleshooting) section.


---

# PostgreSQL as LakeXpress DB

## 1. Create the database[​](#1-create-the-database "Direct link to 1. Create the database")

Run as a PostgreSQL administrator (for example `postgres`):

```
CREATE USER lakexpress_admin WITH PASSWORD 'your_password';
CREATE DATABASE lakexpress_log OWNER lakexpress_admin;
```

LakeXpress creates its tables in the `public` schema. Making `lakexpress_admin` the **owner** of the database gives it the right to create them. `GRANT ALL PRIVILEGES ON DATABASE` alone is not enough on PostgreSQL 15 and later.

## 2. Add it to the credentials file[​](#2-add-it-to-the-credentials-file "Direct link to 2. Add it to the credentials file")

```
{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "pg.company.com",
      "port": 5432,
      "database": "lakexpress_log"
    }
  }
}
```

The connection parameters are the same as for a PostgreSQL source (SSL mode, connection string): see [PostgreSQL connection parameters](/latest/configuration/databases/postgresql.md#connection-parameters).

## 3. Create the LakeXpress tables[​](#3-create-the-lakexpress-tables "Direct link to 3. Create the LakeXpress tables")

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_postgres
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**`permission denied for schema public`:** The user can connect but cannot create tables. Make it the owner of the database (`ALTER DATABASE lakexpress_log OWNER TO lakexpress_admin;`), or grant `CREATE` on the schema: `GRANT CREATE ON SCHEMA public TO lakexpress_admin;` (run while connected to `lakexpress_log`).

**Connection refused or authentication failed:** See the [PostgreSQL troubleshooting](/latest/configuration/databases/postgresql.md#troubleshooting) section.


---

# SQL Server as LakeXpress DB

## 1. Create the database[​](#1-create-the-database "Direct link to 1. Create the database")

Run as a SQL Server administrator (for example with `sqlcmd` or SQL Server Management Studio):

```
CREATE DATABASE lakexpress_log;
GO
CREATE LOGIN lakexpress_admin WITH PASSWORD = 'Your_password1';
GO
USE lakexpress_log;
GO
CREATE USER lakexpress_admin FOR LOGIN lakexpress_admin;
ALTER ROLE db_owner ADD MEMBER lakexpress_admin;
GO
```

LakeXpress creates its tables and sequences in the `dbo` schema, which `db_owner` allows. The password must meet the server's password policy.

## 2. Add it to the credentials file[​](#2-add-it-to-the-credentials-file "Direct link to 2. Add it to the credentials file")

```
{
  "lxdb_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "mssql.company.com",
      "port": 1433,
      "database": "lakexpress_log"
    }
  }
}
```

If the server uses a self-signed certificate (common on test servers and `localhost`), add `"trust_server_certificate": true` to `info`.

The connection parameters are the same as for a SQL Server source (Windows authentication, connection string): see [SQL Server connection parameters](/latest/configuration/databases/sql-server.md#connection-parameters).

## 3. Create the LakeXpress tables[​](#3-create-the-lakexpress-tables "Direct link to 3. Create the LakeXpress tables")

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_mssql
```

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**`User does not have permission to perform this action. (15247)`:** The user can connect but is not a member of `db_owner` in `lakexpress_log`, so it cannot create the LakeXpress tables. Run the `USE lakexpress_log` and `ALTER ROLE` lines again.

**Login failed or cannot open the database:** See the [SQL Server troubleshooting](/latest/configuration/databases/sql-server.md#troubleshooting) section.


---

# SQLite as LakeXpress DB

SQLite needs no server: the LakeXpress DB is a single file. It is the quickest way to try LakeXpress. SQLite can only be used as the LakeXpress DB, not as a source database.

## 1. Add it to the credentials file[​](#1-add-it-to-the-credentials-file "Direct link to 1. Add it to the credentials file")

There is nothing to create beforehand: LakeXpress creates the file, and any missing folder in its path, on first use.

```
{
  "lxdb_sqlite": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/var/lib/lakexpress/lakexpress_log.sqlite"
    }
  }
}
```

| Parameter   | Required | Description             |
| ----------- | -------- | ----------------------- |
| `ds_type`   | Yes      | Must be `"sqlite"`      |
| `auth_mode` | Yes      | Must be `"filesystem"`  |
| `filepath`  | Yes      | Path to the SQLite file |

Use an absolute path. A relative path is resolved against the directory LakeXpress is run from, so running it from another directory would create a new, empty LakeXpress DB there.

## 2. Create the LakeXpress tables[​](#2-create-the-lakexpress-tables "Direct link to 2. Create the LakeXpress tables")

```
./LakeXpress lxdb init -a credentials.json --lxdb_auth_id lxdb_sqlite
```

## When to use SQLite[​](#when-to-use-sqlite "Direct link to When to use SQLite")

**Good for:** trying LakeXpress, development, single-user setups without a database server.

**Avoid for:** several users or machines sharing configurations, syncs running in parallel.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Unable to open database file:** The user running LakeXpress cannot write to the file or to its folder. Check the permissions of the path in `filepath`, and the free disk space.


---

# CLI Commands

| Command                                                        | Purpose                                                     |
| -------------------------------------------------------------- | ----------------------------------------------------------- |
| [`lxdb init`](/latest/cli/lxdb.md#lxdb-init)                   | Create the LakeXpress tables in the LakeXpress DB           |
| [`lxdb drop`](/latest/cli/lxdb.md#lxdb-drop)                   | Drop the LakeXpress tables and their data                   |
| [`lxdb truncate`](/latest/cli/lxdb.md#lxdb-truncate)           | Delete all data from the LakeXpress tables, keep the tables |
| [`lxdb locks`](/latest/cli/lxdb.md#lxdb-locks)                 | List the tables locked by incremental syncs                 |
| [`lxdb release-locks`](/latest/cli/lxdb.md#lxdb-release-locks) | Release locks left by a crashed sync                        |
| [`config create`](/latest/cli/config.md#config-create)         | Create a sync configuration                                 |
| [`config list`](/latest/cli/config.md#config-list)             | List the sync configurations                                |
| [`config delete`](/latest/cli/config.md#config-delete)         | Delete a sync configuration and its run history             |
| [`sync`](/latest/cli/sync.md#sync-command)                     | Run a sync configuration: export, then publish              |
| [`sync[export]`](/latest/cli/sync.md#sync-export)              | Export only                                                 |
| [`sync[publish]`](/latest/cli/sync.md#sync-publish)            | Publish the files of a previous run                         |
| [`status`](/latest/cli/status.md)                              | Show the runs of the sync configurations and their status   |
| [`cleanup`](/latest/cli/cleanup.md)                            | Remove stale or failed runs                                 |
| [`run`](/latest/cli/run.md)                                    | Run a legacy YAML configuration file                        |


---

# Compatibility

## FastBCP Engine[​](#fastbcp-engine "Direct link to FastBCP Engine")

LakeXpress requires FastBCP **1.4.0** or later and is tested with 1.4.0, the version bundled in the package. See [FastBCP version](/latest/installation.md#fastbcp-version).

## Supported Databases[​](#supported-databases "Direct link to Supported Databases")

| Database Type | As Source     | Extraction (default)                | As LakeXpress DB |
| ------------- | ------------- | ----------------------------------- | ---------------- |
| SQL Server    | Supported     | ADBC (`adbc_mssql`)                 | Supported        |
| PostgreSQL    | Supported     | ADBC (`adbc_pgsql`)                 | Supported        |
| Oracle        | Supported     | Native (ADBC with `use_adbc: true`) | Not supported    |
| MySQL         | Supported     | Native                              | Supported        |
| MariaDB       | Supported     | Native                              | Not supported    |
| SAP HANA      | Supported     | Native                              | Not supported    |
| Teradata      | Supported     | ADBC (`adbc_teradata`)              | Not supported    |
| SQLite        | Not supported |                                     | Supported        |
| DuckDB        | Not supported |                                     | Supported        |

ADBC extraction needs FastBCP 1.4.0 or later (bundled) on linux-x64 or win-x64. `"use_adbc": false` on a source selects the native driver, and `"use_adbc": true` selects ADBC for Oracle; see [ADBC Extraction](/latest/configuration/database-configuration.md#adbc-extraction).

## Supported Storage Backends[​](#supported-storage-backends "Direct link to Supported Storage Backends")

| Storage Backend                      | Support Status |
| ------------------------------------ | -------------- |
| **Local Filesystem**                 | Supported      |
| **AWS S3**                           | Supported      |
| **S3-Compatible (OVH, MinIO, etc.)** | Supported      |
| **Google Cloud Storage (GCS)**       | Supported      |
| **Azure Storage (ADLS Gen2, Blob)**  | Supported      |
| **Microsoft Fabric OneLake**         | Supported      |

## Supported Publishing Targets[​](#supported-publishing-targets "Direct link to Supported Publishing Targets")

After export, LakeXpress can create tables in analytics platforms as **external** (data stays in cloud storage) or **internal** (data loaded into native storage).

| Publishing Target    | External  | Internal      |
| -------------------- | --------- | ------------- |
| **Snowflake**        | Supported | Supported     |
| **Databricks**       | Supported | Supported     |
| **Microsoft Fabric** | Supported | Supported     |
| **BigQuery**         | Supported | Supported     |
| **MotherDuck**       | Supported | Supported     |
| **Amazon Redshift**  | Supported | Supported     |
| **AWS Glue**         | Supported | Not supported |
| **DuckLake**         | Supported | Not supported |

Set `--publish_method external` or `--publish_method internal`. Default is `external`, except for Fabric Lakehouse (`internal`).

Publishing guides

Setup, authentication and options for each platform

* [![](/img/logos/snowflake.svg)Snowflake](/latest/integrations/snowflake-publishing.md)
* [![](/img/logos/aws-glue.svg)AWS Glue](/latest/integrations/glue-publishing.md)
* [![](/img/logos/databricks.svg)Databricks](/latest/integrations/databricks-publishing.md)
* [![](/img/logos/microsoft-fabric.png)Microsoft Fabric](/latest/integrations/fabric-publishing.md)
* [![](/img/logos/bigquery.svg)BigQuery](/latest/integrations/bigquery-publishing.md)
* [Amazon Redshift](/latest/integrations/redshift-publishing.md)
* [![](/img/logos/motherduck.svg)MotherDuck](/latest/integrations/motherduck-publishing.md)
* [![](/img/logos/ducklake.svg)DuckLake](/latest/integrations/ducklake-publishing.md)


---

# Exit Codes

LakeXpress returns a structured exit code so that schedulers (cron, Airflow, Control-M, …) can react to the cause of a failure. Codes are grouped by range:

| Range | Meaning                                         |
| ----- | ----------------------------------------------- |
| 0     | Success                                         |
| 1     | Unexpected error                                |
| 10–19 | Configuration and setup — fix the configuration |
| 20–29 | Input validation                                |
| 30–39 | Operational — fix the environment               |
| 40–49 | Execution — some or all work failed             |
| 50–59 | Interruption                                    |

## Codes returned[​](#codes-returned "Direct link to Codes returned")

| Code | Name                          | Meaning                                                                                                                                                                                                                                                                                                       |
| ---- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0    | `SUCCESS`                     | All operations completed successfully.                                                                                                                                                                                                                                                                        |
| 1    | `GENERAL_ERROR`               | Generic or unexpected error.                                                                                                                                                                                                                                                                                  |
| 10   | `CONFIG_ERROR`                | Invalid configuration, malformed credentials file, or missing required setting.                                                                                                                                                                                                                               |
| 11   | `AUTH_ERROR`                  | Authentication or authorization failure.                                                                                                                                                                                                                                                                      |
| 12   | `CONNECTION_ERROR`            | Connection to the source or LakeXpress database failed.                                                                                                                                                                                                                                                       |
| 13   | `LICENSE_ERROR`               | License validation failed (invalid, expired or missing), a source extracted through ADBC has no ADBC driver license, or FastBCP rejected its licence during the export (FastBCP exit code 99 or 100). In the last case the run stops at the first job that fails this way and the other jobs are not started. |
| 21   | `FILE_NOT_FOUND`              | A required file was not found.                                                                                                                                                                                                                                                                                |
| 30   | `FASTBCP_NOT_FOUND`           | FastBCP is not configured (no `--fastbcp_dir_path` and no bundled `engine/` directory) or its executable is missing from the configured directory.                                                                                                                                                            |
| 31   | `INIT_DB_ERROR`               | LakeXpress database initialization, truncation or drop failed.                                                                                                                                                                                                                                                |
| 34   | `FASTBCP_VERSION_UNSUPPORTED` | The FastBCP version is older than the minimum LakeXpress supports (see [Installation](/latest/installation.md#fastbcp-version)).                                                                                                                                                                              |
| 40   | `PARTIAL_FAILURE`             | Some export jobs failed, others succeeded. Resume the run to retry the failed jobs.                                                                                                                                                                                                                           |
| 41   | `ALL_JOBS_FAILED`             | Every export job of the run failed.                                                                                                                                                                                                                                                                           |
| 44   | `NO_TABLES_FOUND`             | No table matched the source schemas and patterns.                                                                                                                                                                                                                                                             |
| 45   | `RUN_NOT_FOUND`               | `--resume` found no run to resume (or the given `--run_id` does not exist), or `sync[publish]` found no completed export run.                                                                                                                                                                                 |
| 46   | `PUBLISH_FAILURE`             | The export succeeded but publishing had failures (`sync`), or publishing failed (`sync[publish]`).                                                                                                                                                                                                            |
| 47   | `EXPORT_AND_PUBLISH_FAILURE`  | Some export jobs failed and publishing had failures.                                                                                                                                                                                                                                                          |
| 50   | `USER_CANCELLED`              | The user interrupted the operation (Ctrl+C).                                                                                                                                                                                                                                                                  |

When every export job fails (41), `sync` does not publish.

## Reserved codes[​](#reserved-codes "Direct link to Reserved codes")

These codes are defined for future use. LakeXpress does not return them yet; schedulers can already treat them by range.

| Code | Name                     | Meaning                                          |
| ---- | ------------------------ | ------------------------------------------------ |
| 20   | `VALIDATION_ERROR`       | Invalid argument or format.                      |
| 22   | `PERMISSION_ERROR`       | File or directory permission error.              |
| 32   | `SCHEMA_DISCOVERY_ERROR` | Table or schema discovery failed.                |
| 33   | `STORAGE_CONFIG_ERROR`   | Storage configuration error (S3, GCS, Azure, …). |
| 42   | `TIMEOUT_ERROR`          | Operation timeout.                               |
| 43   | `JOB_CREATION_FAILED`    | Export jobs could not be created.                |

The FastBCP checks (codes 30 and 34) run before an export (`sync`, `sync[export]`, `run`). `sync[publish]` does not need FastBCP and skips them.


---

# Logging and Monitoring

LakeXpress uses two logging systems: a database for structured export tracking and log files for operational details.

## Database Logging[​](#database-logging "Direct link to Database Logging")

The export tracking database (PostgreSQL, SQL Server, MySQL, SQLite or DuckDB) records all export activity for reporting and troubleshooting.

### Database Schema Overview[​](#database-schema-overview "Direct link to Database Schema Overview")

### Key Tables[​](#key-tables "Direct link to Key Tables")

Tables grouped by role. Click a table in the schema above to jump to its description, and **Show in schema** on a card to go back up.

#### Configuration[​](#configuration "Direct link to Configuration")

export\_syncs

One row per sync configuration, created by config create and read by every sync run.

* `sync_id`, environment name
* Source credentials, database and schemas
* Include and exclude patterns, min and max rows
* Target storage, output directory, sub-path
* Publish target, method, database, schema and table patterns
* FastBCP path, parallelism, number of jobs
* Compression, large-table threshold
* Incremental safety lag
* Manifest, error action, enabled flag
* LakeXpress version, audit timestamps

export\_steps

Workflow step definitions, referenced by jobs, publish jobs and log entries.

* `step_id`
* Step name and description

partition\_columnsnot in the schema

FastBCP partitioning configuration per source table.

* Partition config ID
* Source table (db type, database, schema, table)
* FastBCP distribution key, method, parallel degree
* Environment name, enabled flag
* Description, audit timestamps

#### Export runs and jobs[​](#export-runs-and-jobs "Direct link to Export runs and jobs")

export\_runs

Top-level record of each export run of a sync.

* `run_id`, `sync_id`, run sequence
* Run type: `incremental` when the sync has incremental tables, `full` otherwise
* Source database and schemas
* Storage configuration
* Start and completion times, status
* Total, completed and failed jobs
* Rows exported, storage size
* LakeXpress and FastBCP versions

export\_jobs

One row per table exported in a run.

* `job_id`, `run_id`, `step_id`
* Source db type, database, schema, table, estimated rows
* Storage type, base path, file format, compression
* Parquet row group size
* FastBCP distribution key, method, parallel degree
* Files and rows exported
* Job status, step status, retry count
* Created, started and completed times

#### Files and columns[​](#files-and-columns "Direct link to Files and columns")

export\_files

One row per Parquet file written by a job.

* `file_id`, `job_id`
* File name and full path
* Storage location (local, S3, Azure, GCS)
* File size, row count, status
* Creation time

export\_columns

Column metadata of each exported table. Primary key: job\_id and source column name.

* `job_id`, source column name
* Source db type and data type
* Parquet column name (after transformation)
* Ordinal position, length, precision, scale
* Nullability, primary-key flag

#### Logs[​](#logs "Direct link to Logs")

export\_log

Event log of runs, jobs and steps.

* `log_id`, timestamp
* `run_id`, `job_id`, `step_id`
* Log level (INFO, WARNING, ERROR, DEBUG)
* Event status, elapsed time in seconds
* Source database, schema and table
* Event message, error code and message

#### Incremental sync[​](#incremental-sync "Direct link to Incremental sync")

incremental\_tables

One row per table synced incrementally in a sync, from its --incremental\_table spec.

* `incremental_table_id`, `sync_id`
* Source schema and table
* Incremental column and type (`date`, `datetime`, `int`, `bigint`, `auto`)
* Lower bound inclusive or exclusive, start value
* Loading strategy (`append`, `upsert`, `delete_insert`)
* Mode, publish method, primary key columns
* Enabled flag, description, audit timestamps

incremental\_watermarks

Current watermark of each incremental table: where the next sync starts.

* `watermark_id`, `incremental_table_id`
* Last exported value, `d_new`
* Schema version and hash
* Active run and its start time
* Last export time, last run, rows exported
* Version, update time

incremental\_export\_history

One row per incremental export of a table, with the window it covered.

* `history_id`, `watermark_id`, `run_id`
* Window bounds `d_old` and `d_new`
* Export mode, rows exported, status
* Output path
* Start and completion times
* Replay flag, error message

#### Publishing[​](#publishing "Direct link to Publishing")

publish\_runs

One row per publish run, linked to the export run it publishes.

* `publish_run_id`, `export_run_id`
* Target platform, credentials, database, stage
* Platform configuration, publish options
* Status, start and completion times
* Total, completed and failed jobs
* Schemas, tables and views created or failed
* Rows loaded, error message

publish\_jobs

One row per table published in a publish run.

* `publish_job_id`, `publish_run_id`, `export_job_id`, `step_id`
* Source schema and table
* Target platform, database, schema, table
* Table type, creation and status
* View name, creation and status
* Data loaded, rows loaded, load status
* Job status, retry count, max retries
* Error code and message
* Created, started and completed times, elapsed seconds

## File-Based Logging[​](#file-based-logging "Direct link to File-Based Logging")

Each run creates a log file named: `lx_YYYYMMDD_HHMMSS_<run_id>.log`

### Log Format[​](#log-format "Direct link to Log Format")

Structured format with timestamp, log level, process ID, and context:

```
[YYYY-MM-DD HH:MM:SS.fff+TZ :: LEVEL :: PID :: Context] Message
```

### Sample Log Output[​](#sample-log-output "Direct link to Sample Log Output")

```
[2025-11-05 10:15:23.145+01:00 :: INFO :: 12345 :: MainProcess] Log file path: /var/log/lakexpress/lx_20251105_101523_a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d.log
[2025-11-05 10:15:23.146+01:00 :: INFO :: 12345 :: MainProcess] **** Starting export run ****
[2025-11-05 10:15:23.147+01:00 :: INFO :: 12345 :: MainProcess] Source: SQLServer - salesdb.dbo
[2025-11-05 10:15:23.148+01:00 :: INFO :: 12345 :: MainProcess] Target storage: S3 - s3://data-lake/exports/
[2025-11-05 10:15:23.250+01:00 :: INFO :: 12345 :: MainProcess] Discovered 15 tables for export
[2025-11-05 10:15:23.385+01:00 :: INFO :: 12345 :: Worker-1] Exporting table: orders (estimated 1.2M rows)
[2025-11-05 10:15:45.721+01:00 :: INFO :: 12345 :: Worker-1] Export completed: orders - 1,234,567 rows in 8 files (245 MB)
[2025-11-05 10:15:45.722+01:00 :: INFO :: 12345 :: Worker-1] Export elapsed time: 22.337 seconds
```

### Log Levels[​](#log-levels "Direct link to Log Levels")

Set via `--log_level` (placed after the subcommand, e.g. `sync --log_level DEBUG`). Valid choices are `DEBUG`, `INFO`, `WARNING`, `ERROR` (default `INFO`) -- `CRITICAL` is not an accepted value and the code never emits a `CRITICAL`-level record:

INFOdefault

Progress, row counts, operational messages

WARNING

Non-critical issues (schema differences, type conversions)

ERROR

Failures preventing a table export

DEBUG

SQL queries, file operations, diagnostics

### Log Location[​](#log-location "Direct link to Log Location")

Log files are written one per run, named `lx_<date>_<time>_<run_id>.log`. An export run also saves a copy of its configuration next to it (`lx_<date>_<time>_<run_id>.yml`); `sync[publish]` writes its own log file, named after the export run it publishes. They go to the current working directory by default. Override with `--log_dir` (the directory is created if needed):

```
./LakeXpress sync -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID> --log_dir /var/log/lakexpress
```

### Terminal Output[​](#terminal-output "Direct link to Terminal Output")

Logs also appear in the terminal, formatted with fixed (not severity-based) colors: the timestamp, level tag, process ID, and process name each render in their own consistent color regardless of whether the message is INFO, WARNING, or ERROR. In particular, the level tag is always cyan in standard mode (white in DEBUG mode) -- there is no green/yellow/red success/warning/error color-coding.


---

# Sitemap

All pages of this documentation, in the order of the sidebar.

* [Overview](/latest/.md)
* [Getting Started](/latest/guides/quickstart.md)
* [Command Wizard](/latest/wizard.md)
* [Installation](/latest/installation.md)

## Build Your Pipeline

* 1\. Set Up LakeXpress

  * [Credentials File](/latest/configuration/database-configuration.md)

  * [LakeXpress DB](/latest/lakexpress-db/.md)

    * [PostgreSQL](/latest/lakexpress-db/postgresql.md)
    * [SQL Server](/latest/lakexpress-db/sql-server.md)
    * [MySQL](/latest/lakexpress-db/mysql.md)
    * [SQLite](/latest/lakexpress-db/sqlite.md)
    * [DuckDB](/latest/lakexpress-db/duckdb.md)

* 2\. Connect a Source

  * [PostgreSQL](/latest/configuration/databases/postgresql.md)
  * [SQL Server](/latest/configuration/databases/sql-server.md)
  * [Oracle](/latest/configuration/databases/oracle.md)
  * [MySQL](/latest/configuration/databases/mysql.md)
  * [MariaDB](/latest/configuration/databases/mariadb.md)
  * [SAP HANA](/latest/configuration/databases/sap-hana.md)
  * [Teradata](/latest/configuration/databases/teradata.md)

* [3. Choose Storage](/latest/configuration/storage/storage-overview.md)

  * [Local Filesystem](/latest/configuration/storage/local-filesystem.md)
  * [S3-Compatible Storage](/latest/configuration/storage/s3.md)
  * [Azure Storage](/latest/configuration/storage/azure.md)
  * [Google Cloud Storage (GCS)](/latest/configuration/storage/gcs.md)
  * [Microsoft Fabric OneLake](/latest/configuration/storage/onelake.md)

* 4\. Publish

  * Snowflake

    * [Snowflake Publishing](/latest/integrations/snowflake-publishing.md)
    * [Snowflake Reference](/latest/integrations/snowflake-reference.md)

  * Databricks

    * [Databricks Publishing](/latest/integrations/databricks-publishing.md)
    * [Databricks Reference](/latest/integrations/databricks-reference.md)

  * Microsoft Fabric

    * [Microsoft Fabric Publishing](/latest/integrations/fabric-publishing.md)
    * [Microsoft Fabric Reference](/latest/integrations/fabric-reference.md)

  * BigQuery

    * [BigQuery Publishing](/latest/integrations/bigquery-publishing.md)
    * [BigQuery Reference](/latest/integrations/bigquery-reference.md)

  * Amazon Redshift

    * [Amazon Redshift Publishing](/latest/integrations/redshift-publishing.md)
    * [Amazon Redshift Reference](/latest/integrations/redshift-reference.md)

  * AWS Glue

    * [AWS Glue Publishing](/latest/integrations/glue-publishing.md)
    * [AWS Glue Reference](/latest/integrations/glue-reference.md)

  * MotherDuck

    * [MotherDuck Publishing](/latest/integrations/motherduck-publishing.md)
    * [MotherDuck Reference](/latest/integrations/motherduck-reference.md)

  * [DuckLake](/latest/integrations/ducklake-publishing.md)

* 5\. Tune Your Sync

  * [Table Filtering](/latest/guides/recipes/table-filtering.md)

  * [Parallelism & Performance](/latest/advanced/export-performance.md)

  * [Incremental Sync](/latest/advanced/incremental-sync.md)

    * [Loading Strategies](/latest/advanced/incremental/loading-strategies.md)
    * [Watermarks](/latest/advanced/incremental/watermarks.md)
    * [Incremental Sync](/latest/guides/recipes/incremental-sync.md)
    * [Troubleshooting](/latest/advanced/incremental/troubleshooting.md)

## Monitor & Maintain

* [Logging and Monitoring](/latest/reference/logging.md)
* [Configuration Management](/latest/guides/recipes/configuration-management.md)
* [Resume Failed Exports](/latest/guides/recipes/resume-failed-exports.md)
* [LakeXpress DB](/latest/guides/recipes/lakexpress-db.md)

## Examples

* [Examples](/latest/guides/command-cookbook.md)

* [Local Exports](/latest/guides/recipes/local-exports.md)

* Cloud Storage

  * [AWS S3](/latest/guides/recipes/s3.md)
  * [Azure Storage](/latest/guides/recipes/azure.md)
  * [Google Cloud Storage](/latest/guides/recipes/gcs.md)
  * [OneLake](/latest/guides/recipes/onelake.md)

* Publishing

  * [Snowflake](/latest/guides/recipes/snowflake.md)
  * [Databricks](/latest/guides/recipes/databricks.md)
  * [Microsoft Fabric](/latest/guides/recipes/fabric.md)
  * [BigQuery](/latest/guides/recipes/bigquery.md)
  * [Amazon Redshift](/latest/guides/recipes/redshift.md)
  * [AWS Glue](/latest/guides/recipes/glue.md)
  * [MotherDuck](/latest/guides/recipes/motherduck.md)
  * [DuckLake](/latest/guides/recipes/ducklake.md)

## Reference

* [CLI](/latest/cli/.md)

  * [lxdb](/latest/cli/lxdb.md)
  * [config](/latest/cli/config.md)
  * [sync](/latest/cli/sync.md)
  * [status](/latest/cli/status.md)
  * [cleanup](/latest/cli/cleanup.md)
  * [run](/latest/cli/run.md)

* [Compatibility](/latest/reference/compatibility.md)

* [Exit Codes](/latest/reference/exit-codes.md)

## AI Integrations

* [Agent Skills](/latest/ai/agent-skills.md)
* [MCP Server](/latest/ai/mcp-server.md)


---

# Command Wizard

Build your LakeXpress command step by step using the interactive wizard below.

<!-- -->

1. 1Set UpWindows · SQL Server
2. 2SourceSQL Server
3. 3StorageLocal
4. 4PublishOff
5. 5TuneDefaults
6. 6OptionsDefaults

Step

<!-- -->

1

<!-- -->

of

<!-- -->

6

### Set Up

#### Platform?

Windows

![Linux](/img/tux.svg)Linux

#### Credentials

| Authentication File? : | credentials.json                                                  |
| ---------------------- | ----------------------------------------------------------------- |
| Sync ID (optional)? :  | Auto-generated if empty1-64 chars, alphanumeric/underscore/hyphen |

#### LakeXpress DB

SQL Server

PostgreSQL

MySQL

SQLite

DuckDB

| LakeXpress DB ID? : | e.g., lxdb\_sqlserver |
| ------------------- | --------------------- |

Next:

<!-- -->

Source

### Generated Commands

ShareMulti-line\[x]

1. 1

   Initialize the LakeXpress DB

   **Run once:** creates the LakeXpress tables. Safe to run again.

   ```
   .\LakeXpress.exe lxdb init `
     -a credentials.json `
     --lxdb_auth_id <lxdb_auth_id>
   ```

2. 2

   Create the sync configuration

   **Run once:** saves the configuration in the LakeXpress DB and prints the generated sync ID.

   ```
   .\LakeXpress.exe config create `
     -a credentials.json
   ```

3. 3

   Run the sync

   Exports the tables.

   ```
   .\LakeXpress.exe sync `
     -a credentials.json `
     --lxdb_auth_id <lxdb_auth_id> `
     --sync_id <sync_id printed by config create>
   ```


---