AWS Glue Publishing

Commands use Linux syntax. See Examples & Recipes for the Windows equivalents and the arguments every sync needs.

Export to S3, then register tables in AWS Glue Data Catalog.

PostgreSQL to AWS Glue

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

PostgreSQL to AWS Glue with Custom FastBCP Config

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:TO_CHAR(l_shipdate, 'YYYY'):8;orders:Ctid::4" \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

SQL Server to AWS Glue

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 4 \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lakexpress_{schema}" \
    --publish_table_pattern "{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

SQL Server to AWS Glue with Custom FastBCP Config

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --fastbcp_p 2 \
    --fastbcp_table_config "lineitem:DataDriven:YEAR(l_shipdate):8;orders:Physloc::4" \
    --n_jobs 2 \
    --publish_target glue_catalog \
    --publish_database_name "lakexpress_tpch" \
    --publish_table_pattern "{schema}_{table}"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

SQL Server AdventureWorks to AWS Glue

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target glue_catalog \
    --publish_schema_pattern "adventureworksdw"

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

More Examples

The credentials entries used below, and the AWS Glue setup they rely on, are described in AWS Glue Publishing.

Example 1: Basic Export to Glue

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --publish_target glue_catalog

Creates database EXT_public with all tables from the public schema.

Example 2: Custom Naming with Parallel Execution

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_schema_pattern "lx_{schema}" \
    --publish_table_pattern "{table}" \
    --n_jobs 4

Creates database lx_tpch_1 with tables built in parallel.

Example 3: Literal Database Name

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_database_name analytics

Creates (or reuses) the Glue database named exactly analytics, ignoring --publish_schema_pattern.

Example 4: Date-Based Snapshots

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name sales \
    --target_storage_id s3_01 \
    --publish_target glue_catalog \
    --publish_schema_pattern "sales_{date}" \
    --sub_path "daily/$(date +%Y%m%d)"

Creates database sales_20251210, data stored at s3://bucket/lakexpress/daily/20251210/.