Databricks Publishing

LakeXpress creates Databricks Unity Catalog tables from exported Parquet files. Supports external tables (data stays in S3) and managed Delta tables (data loaded into Databricks).

Prerequisites

1. Databricks Setup

Required Resources:

• Databricks Workspace with Unity Catalog enabled
• SQL Warehouse (for DDL execution)
• Catalog with CREATE SCHEMA and CREATE TABLE permissions
• Storage credential for S3 access (external tables only)

Required Permissions:

• CREATE SCHEMA on the target catalog
• CREATE TABLE on the target schema
• USE CATALOG on the target catalog
• For external tables: access to the external location

2. Storage Credential Setup

For external tables, configure a storage credential for S3 access:

-- In Databricks SQL
CREATE STORAGE CREDENTIAL my_s3_credential
WITH (
    AWS_IAM_ROLE = 'arn:aws:iam::123456789012:role/DatabricksS3AccessRole'
);

CREATE EXTERNAL LOCATION my_s3_location
URL 's3://my-datalake-bucket/lakexpress/'
WITH (STORAGE CREDENTIAL my_s3_credential);

3. Credentials Configuration

Both S3 and Databricks credentials are needed in credentials.json:

{
    "s3_01": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://your-bucket-name/path/to/exports",
            "profile": "your-aws-profile"
        }
    },
    "databricks_prod": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}

Authentication

Databricks uses Personal Access Tokens (PAT):

{
    "databricks_prod": {
        "ds_type": "databricks",
        "auth_mode": "token",
        "info": {
            "host": "your-workspace.cloud.databricks.com",
            "http_path": "/sql/1.0/warehouses/your-warehouse-id",
            "access_token": "your-access-token",
            "catalog": "your-catalog"
        }
    }
}

Configuration Fields:

Field	Description	Required
host	Databricks workspace hostname	Yes
http_path	SQL Warehouse HTTP path	Yes
access_token	Personal Access Token (PAT)	Yes
catalog	Target Unity Catalog	Yes
schema	Default schema	No

To get the HTTP path:

1. Go to Databricks SQL > SQL Warehouses
2. Select your warehouse > Connection details
3. Copy the HTTP path

To create a PAT:

1. Go to User Settings > Developer > Access tokens
2. Generate a new token

Other authentication mode: auth_mode: "oauth" is also supported for OAuth-based authentication, in addition to token.

Table Types

External Tables

Reference Parquet data in S3. Databricks stores only metadata; queries read directly from S3.

• No data loading time or Databricks storage costs
• Data stays in place
• Available right after export

Options for config create:

--publish_target databricks_prod
--publish_method external  # Default

Managed Tables (Delta)

Delta tables loaded into Databricks managed storage via COPY INTO. Enables ACID transactions, time travel, and Z-ordering.

• Faster queries with optimized storage and caching
• Full Delta Lake features
• Better for frequently accessed data

Options for config create:

--publish_target databricks_prod
--publish_method internal  # Databricks maps "internal" to managed Delta tables

Configuration Options

Option	Description	Default
--publish_target ID	Credential ID for Databricks target (required)	-
--publish_schema_pattern PATTERN	Schema naming pattern	EXT_{schema} for external, {schema} for internal
--publish_table_pattern PATTERN	Table naming pattern	{table}
--publish_method {external,internal}	external (Unity Catalog external table) or internal (managed Delta table)	external
--pk_constraints	Add PRIMARY KEY constraints (managed/Delta tables only)	Disabled
--n_jobs N	Parallel workers for table creation	1

There is no --databricks_catalog flag -- the target catalog always comes from the catalog field of the Databricks credential.

Dynamic Naming Patterns

Schema and table names support token-based patterns.

Supported Tokens

Token	Description	Example Output
{schema}	Source schema name	tpch_1
{table}	Source table name	customer
{database}	Source database name	tpch
{date}	Current date (YYYYMMDD)	20251210
{timestamp}	Current timestamp (YYYYMMDD_HHMMSS)	20251210_143022
{uuid}	UUID4, consistent per run	a1b2c3d4-...
{subpath}	CLI --sub_path value	staging

Common Patterns

Prefixed Schemas

--publish_schema_pattern "bronze_{schema}"
--publish_table_pattern "{table}"
--publish_target databricks_prod

# Result:
# lakexpress_catalog.bronze_tpch_1.customer
# lakexpress_catalog.bronze_tpch_1.orders

Date-Based Schemas

--publish_schema_pattern "{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target databricks_prod

# Result:
# lakexpress_catalog.tpch_1_20251210.customer

Medallion Architecture

# Bronze layer (raw data, external tables)
--publish_schema_pattern "bronze_{schema}"
--publish_method external
--publish_target databricks_prod

# Silver layer (curated data, managed Delta tables)
--publish_schema_pattern "silver_{schema}"
--publish_method internal
--publish_target databricks_prod

Usage Examples

Ready-to-run commands for Databricks are in Examples & Recipes: Databricks.

See Also

• Databricks Reference - Data type mappings, CLI arguments, querying, Delta features
• CLI Reference - All command-line options
• Examples & Recipes - Working command examples