Azure Storage

Supports Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage via Service Principal authentication.

Configuration

{
  "azure_storage": {
    "ds_type": "azure",
    "auth_mode": "service_principal",
    "info": {
      "directory": "abfss://your-container@your-storage-account.dfs.core.windows.net/path/to/exports",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}

note

Use $env{VAR_NAME} syntax to reference environment variables. Set the variable before running LakeXpress (e.g., export LX_AZURE_CLIENT_SECRET="your-secret").

Field	Required	Description
ds_type	Yes	Must be "azure"
auth_mode	Yes	Must be "service_principal"
directory	Yes	Azure storage URL (abfss:// for ADLS Gen2, abs:// for Blob)
azure_client_id	Yes	Application (client) ID from Azure AD app registration
azure_tenant_id	Yes	Directory (tenant) ID from Azure AD
azure_client_secret	Yes	Client secret value from Azure AD app

Storage URL Formats

ADLS Gen2:

abfss://container-name.dfs.core.windows.net/path/to/exports

Azure Blob Storage:

abs://container-name.blob.core.windows.net/path/to/exports

The URL format uses .dfs.core.windows.net (not @storageaccount.dfs.core.windows.net).

Azure Authentication Setup

1. Create an Azure AD Application (Service Principal):

In Azure Portal App Registrations:

• Click "New registration"
• Note the Application (client) ID and Directory (tenant) ID
• Go to "Certificates & secrets" -> "New client secret"
• Copy the client secret value (shown only once)

2. Assign Storage Permissions:

In your Storage Account -> Access Control (IAM), add the "Storage Blob Data Contributor" role to your service principal.

Azure RBAC roles reference

Usage

./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id azure_01 \
        --fastbcp_dir_path /path/to/fastbcp

Path Structure

ADLS Gen2:

abfss://container@storage.dfs.core.windows.net/path/to/exports/schema_name/table_name/part-00000.parquet

With --sub_path:

abfss://container@storage.dfs.core.windows.net/path/to/exports/sub_path/schema_name/table_name/part-00000.parquet

Required Permissions

The Service Principal needs Storage Blob Data Contributor, which grants:

• Microsoft.Storage/storageAccounts/blobServices/containers/read
• Microsoft.Storage/storageAccounts/blobServices/containers/write
• Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action

Troubleshooting

Problem	Solution
Authentication / access denied	Verify client ID, tenant ID, and secret. Confirm "Storage Blob Data Contributor" role. Check network access rules. Verify container exists.
Invalid URL format	Use abfss:// for ADLS Gen2, abs:// for Blob. Format: abfss://container@storageaccount.dfs.core.windows.net/path.
Connection timeout	Check firewall rules, storage account firewall allowlist, and private endpoint restrictions.