S3-Compatible Storage

All S3-compatible providers use ds_type: "s3". Provider differentiation is handled through endpoint_url in the AWS profile.

Supported providers: Amazon S3, OVH S3, MinIO, Wasabi, DigitalOcean Spaces, Backblaze B2, and any S3-compatible object storage.

Method 1: AWS Profile (Recommended for AWS)

Uses ~/.aws/config and ~/.aws/credentials:

{
  "s3_01": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-bucket-name/lakexpress/exports",
      "profile": "your-aws-profile"
    }
  }
}

~/.aws/config:

[profile your-aws-profile]
region = us-east-1

~/.aws/credentials:

[your-aws-profile]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Field	Required	Description
`ds_type`	Yes	Must be `"s3"`
`auth_mode`	Yes	Must be `"profile"`
`directory`	Yes	Full S3 URL (e.g., `s3://bucket/path`)
`profile`	Yes	AWS profile name from `~/.aws/credentials`

Method 2: S3-Compatible Providers (OVH, MinIO, etc.)

Use an AWS profile with a custom endpoint_url.

OVH S3:

{
  "s3_02": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://your-ovh-bucket/lakexpress",
      "profile": "ovh"
    }
  }
}

[profile ovh]
endpoint_url = https://s3.gra.io.cloud.ovh.net
region = gra

MinIO:

{
  "s3_03": {
    "ds_type": "s3",
    "auth_mode": "profile",
    "info": {
      "directory": "s3://data-lake/exports",
      "profile": "minio"
    }
  }
}

[profile minio]
endpoint_url = http://localhost:9000
region = us-east-1

Field	Required	Description
`ds_type`	Yes	Must be `"s3"`
`auth_mode`	Yes	Must be `"profile"`
`directory`	Yes	Full S3 URL (e.g., `s3://bucket/path`)
`profile`	Yes	AWS profile name (with `endpoint_url` in `~/.aws/config`)

Usage

# AWS S3
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_01 \
        --fastbcp_dir_path /path/to/fastbcp

# OVH S3
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_02 \
        --fastbcp_dir_path /path/to/fastbcp

# MinIO
./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id s3_03 \
        --fastbcp_dir_path /path/to/fastbcp

Path Structure

s3://bucket-name/base_path/schema_name/table_name/part-00000.parquet

With --sub_path:

s3://bucket-name/base_path/sub_path/schema_name/table_name/part-00000.parquet

Required IAM Permissions (AWS S3)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:s3:::your-bucket-name"
      ]
    }
  ]
}

Backward Compatibility

Custom storage types ending in _s3 (e.g., minio_s3, custom_s3) still work, but storage_type: "s3" is preferred.

Troubleshooting

AWS S3

Problem	Solution
Access denied / credential errors	Verify credentials: `aws s3 ls s3://your-bucket --profile your-profile`. Check IAM includes `s3:PutObject`. Verify region.
Slow uploads	Enable S3 Transfer Acceleration. Use S3 in the same region as your source database.