BigQuery Publishing

Commands use Linux syntax. See Examples & Recipes for the Windows equivalents and the arguments every sync needs.

The credentials entries used below, and the BigQuery setup they rely on, are described in BigQuery Publishing.

Example 1: External Tables (Default)

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target bigquery_prod

Creates dataset EXT_public (the default schema pattern for the default external publish method) with external tables pointing to GCS Parquet files. Pass --publish_schema_pattern "{schema}" explicitly if you want the dataset named public instead.

Example 2: Native Tables with Parallel Execution

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name tpch \
    --source_schema_name tpch_1 \
    --target_storage_id gcs_01 \
    --sub_path mydata \
    --publish_target bigquery_prod \
    --publish_method internal \
    --publish_schema_pattern "lx_{schema}" \
    --n_jobs 4

Creates dataset lx_tpch_1 with native tables loaded from GCS.

Example 3: Daily Snapshots

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name sales \
    --target_storage_id gcs_01 \
    --sub_path "daily/$(date +%Y%m%d)" \
    --publish_target bigquery_prod \
    --publish_schema_pattern "sales_{date}"

Creates dataset sales_20251210, data stored in gs://bucket/lakexpress/daily/20251210/.

Example 4: EU Data Residency

{
    "bigquery_eu": {
        "ds_type": "bigquery",
        "auth_mode": "profile",
        "info": {
            "project_id": "my-eu-project",
            "location": "EU",
            "profile": "/path/to/service-account.json"
        }
    }
}

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_db_name mydb \
    --source_schema_name gdpr_data \
    --target_storage_id gcs_eu \
    --sub_path mydata \
    --publish_target bigquery_eu

Example 5: With Primary Key Constraints

Add --pk_constraints to include primary key definitions in native (internal) table DDL:

./LakeXpress config create \
    -a credentials.json \
    --lxdb_auth_id lxdb_postgres \
    --source_db_auth_id source_postgres \
    --source_schema_name public \
    --target_storage_id gcs_01 \
    --publish_method internal \
    --publish_target bigquery_prod \
    --pk_constraints