Export Performance

LakeXpress auto-detects the best export method per table. Override when needed using CLI flags or the partition_columns table in the LakeXpress DB.

Export Methods

Without Key Column

NoneAll databases

Sequential export, no parallelism

CtidPostgreSQL

Tuple ID-based parallel export

RowidOracle

Row ID-based parallel export

PhyslocSQL Server

Physical location-based parallel export

With Key Column

Random

Distributes using column values

Key columnInteger or bigint

DataDriven

Splits on distinct values

Key columnColumn with distinct values, e.g. year, region

RangeId

Range partitioning via min/max

Key columnNumeric or date column

Ntile

Even distribution

Key columnNumeric column

Timepartition

Partitions by a date/time column into Hive-style paths

Key columnDate or datetime column, e.g. (col,year,month)

Timepartition

Splits data into time-based chunks where each parallel thread handles a specific time period. Works with any source database -- ideal for time-series data and large historical tables.

Key column format -- a parenthesized list of a date column followed by granularity levels:

(col, year)year=YYYY

(col, year, month)year=YYYY/month=MM

(col, year, month, day)year=YYYY/month=MM/day=DD

(col, year, week)year=YYYY/week=WW

Exported files follow Hive-style partitioned paths, e.g. year=2008/month=01/CARRIER_CLAIMS.parquet. Rows whose date is NULL go to a <col>=NULL/ directory, e.g. CLM_FROM_DT=NULL/CARRIER_CLAIMS.parquet.

Tuning Parameters

--n_jobsdefault 1

Tables exported simultaneously

--fastbcp_por-pdefault 1

Parallel threads per large table

--large_table_thresholddefault 100000

Row count below which tables export sequentially

--compression_typedefault Zstd

Parquet compression: Zstd, Snappy, Gzip, Lz4 or None

Both --n_jobs and --fastbcp_p default to 1 (sequential) — increase them explicitly to enable parallelism. --fastbcp_p only takes effect when its value is greater than 1; tables below --large_table_threshold use the None (sequential) method regardless of --fastbcp_p, unless a per-table method is set (see Per-Table CLI Override and Per-Table Database Override): a per-table method applies whatever the row count.

Parallel degree guidelines (--fastbcp_p, threads per table):

1

< 100K rows

Default. Below the large-table threshold, parallel export is skipped

2–4

100K–10M rows

4–8

10M–100M rows

8–16

> 100M rows

Incremental Tables

An incremental table (--incremental_table) is exported with a query that filters on its watermark column, for the first sync as for the later ones. The methods without a key column (Ctid, Rowid, Physloc) need a whole table and cannot be used with a query: LakeXpress replaces them with None and logs a warning, whether it chose the method itself or you set it with --fastbcp_table_config. A large PostgreSQL, Oracle or SQL Server incremental table is therefore exported with a single thread by default.

To keep parallelism, set a method with a key column for that table:

--incremental_table "fact.transactions:txn_date:date" \
--fastbcp_table_config "fact.transactions:Ntile:txn_id:8"

Per-Table CLI Override

--fastbcp_table_config "[schema.]table:method:key_column:parallel_degree"

The schema prefix is optional. When omitted, the config applies to all schemas containing that table (a warning is logged). Separate several tables with semicolons, or repeat --fastbcp_table_config.

The parallel degree is required and must be an integer: there is no default. An entry without a valid degree is ignored, without an error. The method and key column can be left empty (e.g. orders:::4) to keep the auto-detected method with the given degree. A database-specific method used with another database (e.g. Ctid on SQL Server) is replaced by the right one for that database, with a warning.

# Schema-qualified (targets specific schema)
--fastbcp_table_config "dbo.lineitem:DataDriven:YEAR(l_shipdate):8;dbo.orders:Ctid::4"

# Without schema (applies to all schemas, with warning)
--fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Ctid::4"

# Timepartition -- key column uses parenthesized format
--fastbcp_table_config "CARRIER_CLAIMS:Timepartition:(CLM_FROM_DT,year,month):4"

Per-Table Database Override

Insert into the partition_columns table for persistent configuration:

INSERT INTO partition_columns (
    source_db_type, source_database, source_schema, source_table,
    fastbcp_method, fastbcp_parallel_degree, fastbcp_distribution_key
) VALUES (
    'postgres', 'analytics', 'public', 'events',
    'Ctid', 4, NULL
);

A row only applies to syncs created with the same --env_name (column env_name, default 'default') and while is_enabled is true (the default). Set env_name in the INSERT if your sync uses another environment name.

Database configuration takes priority over CLI configuration, which takes priority over auto-detection.

Monitoring

The real table is export_jobs (there is no plain jobs table), with columns job_status, started_at/completed_at, and num_rows_exported:

SELECT source_table, job_status, started_at, completed_at, num_rows_exported
FROM export_jobs
WHERE run_id = 'your-run-id'
ORDER BY started_at DESC;