DuckLake Publishing

LakeXpress registers exported Parquet files as tables in a DuckLake catalog without copying data -- the catalog stores metadata (schema, file locations, statistics) while the Parquet files stay in place.

Table of Contents

• What is DuckLake?
• Prerequisites
• Catalog Configuration
• Usage Example
• Querying DuckLake Tables
• Notes and Limitations
• See Also

What is DuckLake?

DuckLake is an open table format and DuckDB extension that stores table metadata (schema, snapshots, file listings, statistics) in a SQL catalog database, while the actual data lives as Parquet files in local or cloud storage. It provides ACID transactions, time travel, and schema evolution on top of plain Parquet -- without requiring a dedicated metadata service.

LakeXpress publishes to DuckLake by creating one table per exported source table and registering the corresponding Parquet files with the catalog, so the data can be queried with standard SQL through DuckDB.

Prerequisites

1. The ducklake DuckDB extension

LakeXpress runs INSTALL ducklake and LOAD ducklake automatically before attaching the catalog -- this is a core DuckDB extension (not a community extension), so no manual setup is required as long as the machine running LakeXpress can install it (or already has it cached).

2. A catalog database

DuckLake stores its metadata in a separate database, which LakeXpress calls the catalog. The catalog credential entry in credentials.json must use one of the supported catalog backends: sqlite, duckdb, postgres, or mysql.

3. An intermediate storage location for the Parquet files

The exported Parquet files (local directory or cloud storage such as S3, GCS, or Azure) that were produced by the export step -- referenced via --target_storage_id or --output_dir -- are the files registered into the DuckLake catalog.

Catalog Configuration

The credential entry pointed to by --publish_target is the DuckLake catalog: its ds_type must be one of the four supported catalog backends. LakeXpress detects a DuckLake target automatically when ds_type is sqlite, duckdb, postgres, or mysql (or when the entry explicitly sets "catalog_type": "ducklake").

SQLite catalog

{
    "ducklake_catalog": {
        "ds_type": "sqlite",
        "info": {
            "filepath": "/path/to/ducklake_catalog.sqlite"
        }
    }
}

DuckDB catalog

{
    "ducklake_catalog": {
        "ds_type": "duckdb",
        "info": {
            "filepath": "/path/to/ducklake_catalog.duckdb"
        }
    }
}

PostgreSQL catalog

{
    "ducklake_catalog": {
        "ds_type": "postgres",
        "info": {
            "server": "my-postgres-host",
            "port": 5432,
            "database": "ducklake_meta",
            "username": "ducklake_user",
            "password": "secret",
            "schema": "my_schema"
        }
    }
}

MySQL catalog

{
    "ducklake_catalog": {
        "ds_type": "mysql",
        "info": {
            "server": "my-mysql-host",
            "port": 3306,
            "database": "ducklake_meta",
            "username": "ducklake_user",
            "password": "secret"
        }
    }
}

Field	Backends	Required	Description
ds_type	all	Yes	One of sqlite, duckdb, postgres, mysql
info.filepath	sqlite, duckdb	No	Path to the catalog file. Defaults to ducklake_catalog.sqlite / ducklake_catalog.duckdb in the working directory
info.server	postgres, mysql	No	Host name. Default: localhost
info.port	postgres, mysql	No	Port. Default: 5432 (postgres) / 3306 (mysql)
info.database	postgres, mysql	No	Database name. Default: postgres (postgres backend only)
info.username / info.password	postgres, mysql	Yes	Catalog database credentials
info.schema	postgres only	No	Schema used for the catalog's own metadata tables (set via search_path). Ignored for mysql and sqlite; duckdb handles schema at query time. Not passed for the other backends

Usage Example

LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id log_db \
  --source_db_auth_id source_db \
  --source_db_name MY_DB \
  --source_schema_name MY_SCHEMA \
  --target_storage_id s3_01 \
  --sub_path mydata \
  --publish_target ducklake_catalog \
  --publish_schema_pattern "{schema}" \
  --publish_table_pattern "{table}"

• --publish_target is the credential ID of the DuckLake catalog entry (see Catalog Configuration above).
• --target_storage_id (or --output_dir for local exports) is the storage location holding the exported Parquet files; if it points at cloud storage (S3/GCS/Azure), LakeXpress generates a CREATE SECRET statement so DuckDB can read the files.
• --publish_schema_pattern / --publish_table_pattern follow the same token syntax ({schema}, {table}, {database}, {date}, {timestamp}, {uuid}, {subpath}) as the other publishers. Default: {schema} / {table}.
• --publish_method has no effect on DuckLake: it always registers the exported Parquet files by reference (never copies data into the catalog). It still influences the CLI's generic default schema pattern ({schema} for internal, EXT_{schema} for the default external) unless --publish_schema_pattern is set explicitly.

Querying DuckLake Tables

Once published, query the tables with DuckDB using the same catalog connection string used for publishing:

INSTALL ducklake;
LOAD ducklake;

ATTACH 'sqlite:/path/to/ducklake_catalog.sqlite' AS lakexpress_lake (TYPE ducklake);

SELECT * FROM lakexpress_lake.my_schema.my_table LIMIT 10;

For a PostgreSQL catalog, the attach string instead looks like:

ATTACH 'postgres:dbname=ducklake_meta host=my-postgres-host port=5432 user=ducklake_user password=secret'
    AS lakexpress_lake (TYPE ducklake);

Notes and Limitations

• No data copy: DuckLake tables are registered against the existing exported Parquet files (CALL ducklake_add_data_files(...)) -- the data itself is never moved or duplicated into the catalog database.
• Supported catalog backends: sqlite, duckdb, postgres, and mysql only. Any other ds_type on the --publish_target credential is rejected.
• Default schema name: if the catalog credential does not set info.schema, LakeXpress logs and tracks the catalog schema as main (used for run tracking metadata; it does not affect the target schema/table naming, which is controlled by --publish_schema_pattern / --publish_table_pattern).
• Table replacement: if a target table already exists, LakeXpress drops and recreates it before re-registering files (matching the behavior of the other publishers).
• Column types: when the exported Parquet file is reachable, LakeXpress derives column types directly from the file (DESCRIBE SELECT * FROM read_parquet(...)); otherwise it falls back to mapping the source database types to DuckDB types from the export metadata.
• Sequential publishing: table creation currently runs sequentially regardless of --n_jobs.

See Also

• Storage Configuration Overview - Configuring S3, GCS, Azure, and local storage
• CLI Reference - All command-line options
• MotherDuck Publishing - Sibling DuckDB-family publishing target
• MotherDuck Reference - Data type mappings and querying patterns