Google Cloud Storage (GCS)

Configuration

{
  "gcs_storage": {
    "ds_type": "gcs",
    "auth_mode": "profile",
    "info": {
      "directory": "gs://your-bucket-name/path/to/exports",
      "profile": "/path/to/service-account-key.json"
    }
  }
}

Field	Required	Description
ds_type	Yes	Must be "gcs"
auth_mode	Yes	Must be "profile"
directory	Yes	Full GCS URL (e.g., gs://bucket/path)
profile	Yes	Path to GCS service account JSON key file

Usage

./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id gcs_01 \
        --fastbcp_dir_path /path/to/fastbcp

Path Structure

gs://bucket-name/base_path/schema_name/table_name/part-00000.parquet

Service Account Permissions

Required roles:

• Storage Object Creator
• Storage Object Viewer (if reading is needed)

Or custom IAM permissions:

• storage.objects.create
• storage.objects.delete (if overwriting)
• storage.buckets.get

Creating a Service Account

gcloud iam service-accounts create lakexpress-export \
    --display-name="LakeXpress Export Service Account"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectCreator"

gcloud iam service-accounts keys create service-account-key.json \
    --iam-account=lakexpress-export@YOUR_PROJECT_ID.iam.gserviceaccount.com

Troubleshooting

Problem	Solution
Authentication errors	Verify key file path. Check Storage Object Creator role. Ensure service account is enabled.
Quota exceeded	Check GCS quotas in Google Cloud Console. Request increase if needed.