config

Manage sync configurations. A sync configuration says what to export, where, and where to publish it. It is stored in the LakeXpress DB and run with sync. Examples: Configuration Management recipes, and the other Examples & Recipes.

All config commands also accept the common options.

config create

Create a sync configuration. The command prints its sync_id and the sync command to run it.

usage: LakeXpress config create --auth_file AUTH_FILE
                                --lxdb_auth_id ID
                                --source_db_auth_id ID [OPTIONS]
                                [common options]

At least one of --output_dir (local disk) or --target_storage_id (cloud storage) is required.

Sync Identity

Option	Description
--sync_id ID	Name of the sync configuration: 1 to 64 letters, digits, _ or - (e.g. prod-daily-export). Generated if omitted. Fails if the ID already exists
--env_name NAME	Environment name, to keep separate configurations for dev, test and prod (default: default)

Source

Option	Description
--source_db_auth_id ID	Required. Entry of the credentials file for the source database
--source_db_name NAME	Source database name. Default: info.database of the source credentials entry
--source_schema_name PATTERN	Schemas to export, comma-separated, SQL LIKE patterns allowed (e.g. public, prod_%). Default: all schemas

Table Filtering

Option	Description
-i, --include PATTERN	Only export the tables matching these patterns, comma-separated (e.g. orders%, customer%)
-e, --exclude PATTERN	Skip the tables matching these patterns, comma-separated (e.g. temp%, test%)
--min_rows N	Skip tables with fewer than N rows
--max_rows N	Skip tables with more than N rows

Patterns use SQL LIKE syntax: % matches any characters, _ matches one character.

Incremental Sync

Option	Description
--incremental_table SPEC	Export only new rows of a table, tracked with a watermark column. Repeat for each table. Format: schema.table:column:type[:i|:e][@start][!strategy], e.g. tpch_1.orders:o_orderdate:date
--incremental_safety_lag SECONDS	Stay this many seconds behind the current time, for late-arriving rows (default: 0)

Column types: date, datetime (also for timestamp columns), int, bigint, auto. Lower bound: :e exclusive, column > watermark (default), or :i inclusive, column >= watermark. Tables without --incremental_table are fully exported on each sync. Details: Incremental Sync.

Output

Option	Description
--output_dir PATH	Local directory for the Parquet files. A relative path is resolved against the current directory and saved as an absolute path (0.4.9+; use an absolute path with earlier versions)
--target_storage_id ID	Entry of the credentials file for the cloud storage (S3, GCS, Azure, OneLake)
--sub_path PATH	Folder added between the base path and the schema folders, e.g. staging/temp gives base/staging/temp/schema/table/

Export Engine

Option	Description
--fastbcp_dir_path PATH	Folder containing FastBCP. Not needed with the trial edition, which finds its bundled engine
--n_jobs N	Number of tables exported in parallel (default: 1)
-p, --fastbcp_p N	Parallel processes used by FastBCP within one large table (default: 1)
--large_table_threshold N	Row count above which a table is exported in parallel (default: 100000)
--fastbcp_table_config CONFIG	Per-table FastBCP settings: [schema.]table:method:key_column:degree, separated by ;, e.g. dbo.lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4, or ORDERS:Timepartition:(ORDER_DATE,year,month):8
--compression_type TYPE	Parquet compression: Zstd (default), Snappy, Gzip, Lz4 or None

With --n_jobs 4 --fastbcp_p 2, four tables are exported at the same time, each with two processes.

Publishing

Option	Description
--publish_target ID	Entry of the credentials file for the target platform (Snowflake, Databricks, AWS Glue, Fabric, BigQuery, Redshift, MotherDuck, DuckLake)
--publish_method METHOD	external (default): tables read the Parquet files in cloud storage. internal: data is loaded into the platform
--publish_database_name NAME	Target database name (AWS Glue, Databricks)
--publish_schema_pattern PATTERN	Target schema name, with tokens {schema}, {table}, {database}, {date}, {timestamp}, {uuid}, {subpath}. Default: EXT_{schema} (external), {schema} (internal)
--publish_table_pattern PATTERN	Target table name, same tokens, must include {table}. Default: {table}
--no_views	Do not create views over external tables (Snowflake)
--pk_constraints	Add the source primary keys to internal tables (Snowflake, Databricks, Fabric, BigQuery, MotherDuck, Redshift)

Platform-specific defaults and behavior are described on each target platform page.

Metadata

Option	Description
--generate_metadata	Write CDM metadata (manifest.json and .cdm.json files) next to the Parquet files
--manifest_name NAME	Name of the CDM manifest. Default: schema name (per schema) or database name (global)

Error Handling

Option	Description
--error_action ACTION	fail (default): stop at the first failed table. continue: log the error and go on with the other tables

config list

List the sync configurations.

usage: LakeXpress config list --auth_file AUTH_FILE
                              --lxdb_auth_id ID [--env_name NAME]
                              [common options]

Option	Description
--env_name NAME	Only list the configurations of this environment

config delete

Delete a sync configuration with its run history, table metadata and watermarks. Exported files and published tables are not deleted.

usage: LakeXpress config delete --auth_file AUTH_FILE
                                --lxdb_auth_id ID --sync_id SYNC_ID
                                [--confirm] [common options]

Option	Description
--sync_id ID	Required. Sync configuration to delete
--confirm	Delete it. Without it, shows what would be deleted