Databricks Publishing

Commands use Linux syntax. See Examples & Recipes for the Windows equivalents and the arguments every sync needs.

Export to S3, then create Databricks Unity Catalog tables.

PostgreSQL to Databricks (External)

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_method external

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

PostgreSQL to Databricks (Internal/Managed Delta Tables)

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

SQL Server to Databricks (Internal)

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --source_db_auth_id source_mssql \
    --source_db_name adventureworksdw \
    --source_schema_name dbo \
    --fastbcp_p 2 \
    --n_jobs 4 \
    --target_storage_id s3_02 \
    --publish_target databricks_prod \
    --publish_schema_pattern "adventureworksdw" \
    --publish_method internal

./LakeXpress sync \
    -a credentials.json \
    --lxdb_auth_id lxdb_mssql \
    --sync_id <SYNC_ID>

More Examples

The credentials entries used below, and the Databricks setup they rely on, are described in Databricks Publishing.

Example 1: External Tables (Data in S3)

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_method external

Result: External Parquet tables in lakexpress_catalog.EXT_public reading from S3 (default schema pattern EXT_{schema} for external tables).

Example 2: Managed Delta Tables

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_method internal \
    --n_jobs 4

Result: Delta tables in lakexpress_catalog.tpch_1 loaded via COPY INTO.

Example 3: Custom Schema Pattern with Primary Keys

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name sales \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --publish_schema_pattern "raw_{schema}" \
    --publish_method internal \
    --pk_constraints

Result: Managed Delta tables in {catalog}.raw_sales (customer, orders, etc.) with PRIMARY KEY constraints, where {catalog} is the value of the catalog field in the databricks_prod credential -- there is no --databricks_catalog override flag.

Example 4: Parallel Table Creation

./LakeXpress config create -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name tpch_1 \
    --target_storage_id s3_01 \
    --publish_target databricks_prod \
    --n_jobs 8