Microsoft Fabric OneLake

Exports Parquet files to OneLake via Service Principal authentication.

Configuration

{
  "onelake_storage": {
    "ds_type": "onelake",
    "auth_mode": "service_principal",
    "info": {
      "directory": "onelake://your-workspace-name/your-lakehouse-name/",
      "azure_client_id": "your-application-client-id",
      "azure_tenant_id": "your-directory-tenant-id",
      "azure_client_secret": "$env{LX_AZURE_CLIENT_SECRET}"
    }
  }
}

Field	Required	Description
ds_type	Yes	Must be "onelake"
auth_mode	Yes	Must be "service_principal"
directory	Yes	onelake://workspace/lakehouse/path/
azure_client_id	Yes	Application (client) ID from Azure AD
azure_tenant_id	Yes	Directory (tenant) ID from Azure AD
azure_client_secret	Yes	Client secret value from Azure AD app

URL Format

onelake://workspace-name/lakehouse-name/path/to/exports/

Example:

onelake://FABRICPOC/lakexpress_lakehouse/exports/

Usage

./LakeXpress -a auth.json --lxdb_auth_id lxdb_ms \
        --source_db_auth_id ds_03_pg \
        --target_storage_id fabric_onelake \
        --fastbcp_dir_path /path/to/fastbcp

With config creation:

python lxpress.py config create \
       -a auth.json \
       --lxdb_auth_id lxdb_ms \
       --source_db_auth_id ds_04_pg \
       --source_db_name tpch \
       --source_schema_name tpch_1 \
       --fastbcp_dir_path /path/to/fastbcp \
       --fastbcp_p 2 \
       --n_jobs 4 \
       --target_storage_id fabric_onelake \
       --generate_metadata

Path Structure

onelake://workspace/lakehouse/base_path/schema_name/table_name/part-00000.parquet

With --sub_path:

onelake://workspace/lakehouse/base_path/sub_path/schema_name/table_name/part-00000.parquet

Authentication Setup

1. Create an Azure AD Application (Service Principal):

In Azure Portal App Registrations:

• Click "New registration"
• Note the Application (client) ID and Directory (tenant) ID
• Go to "Certificates & secrets" -> "New client secret"
• Copy the client secret value (shown only once)

2. Grant OneLake Access in Microsoft Fabric:

In the Fabric portal, go to Workspace settings -> "Manage access" and add the service principal with Contributor or Admin permissions. For finer control, grant access at the Lakehouse level.

Troubleshooting

Problem	Solution
Authentication / access denied	Verify client ID, tenant ID, and secret. Confirm Contributor role on workspace. Verify OneLake is enabled.
Invalid URL format	Use onelake:// prefix. Format: onelake://workspace-name/lakehouse-name/path/. Ensure path ends with /.
Connection timeout	Check firewall rules and verify Fabric workspace is network-accessible.