Table Filtering

Commands use Linux syntax. See Examples & Recipes for the Windows equivalents and the arguments every sync needs.

Include Specific Table Patterns

Export only fact and dimension tables:

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --include "fact_%, dim_%" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>

Includes fact_sales, fact_orders, dim_customer, dim_product, etc.

Exclude Temporary and Test Tables

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --exclude "temp_%, test_%, staging_%" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>

Combine Include and Exclude

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name analytics \
    --include "report_%, dashboard_%" \
    --exclude "%_test, %_backup" \
    --output_dir /data/exports \
    --n_jobs 4

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>

Includes report_sales, dashboard_kpi. Excludes report_sales_test, dashboard_kpi_backup.

Filter Schemas and Tables, then Publish with Custom Names

Export the fact_ and dim_ tables of every sales and marketing schema, skip temporary tables, and publish them to Snowflake under one schema:

./LakeXpress config create \
  -a credentials.json \
  --lxdb_auth_id lxdb_postgres \
  --source_db_auth_id source_postgres \
  --source_db_name analytics \
  --source_schema_name "sales%, marketing%" \
  --include "fact_%, dim_%" \
  --exclude "temp%, test%" \
  --target_storage_id s3_01 \
  --sub_path data-lake/prod \
  --publish_target snowflake_prod \
  --publish_schema_pattern "ANALYTICS_{subpath}" \
  --publish_table_pattern "{schema}_{table}" \
  --n_jobs 8 \
  --fastbcp_p 4 \
  --generate_metadata

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --sync_id <SYNC_ID>