AWS Glue Publishing

LakeXpress creates AWS Glue Data Catalog tables from exported Parquet files, enabling queries via Athena, Redshift Spectrum, and EMR.

Prerequisites

1. AWS Glue Permissions

Required IAM permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateDatabase",
                "glue:GetDatabase",
                "glue:DeleteDatabase",
                "glue:CreateTable",
                "glue:GetTable",
                "glue:DeleteTable",
                "glue:UpdateTable",
                "glue:GetTables",
                "glue:BatchCreatePartition",
                "glue:GetPartitions"
            ],
            "Resource": "*"
        }
    ]
}

2. S3 Storage Setup

Both S3 and Glue credentials are required in credentials.json:

{
    "s3_01": {
        "ds_type": "s3",
        "auth_mode": "profile",
        "info": {
            "directory": "s3://my-datalake-bucket/lakexpress/",
            "profile": "my-aws-profile"
        }
    },
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}

tip

Both "aws_glue" and "glue" are accepted as the ds_type value.

Critical: The S3 bucket must be accessible from the Glue Data Catalog in the specified region.

Authentication Modes

Profile (Recommended)

Uses AWS CLI credentials from ~/.aws/credentials:

{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1"
        }
    }
}

• No secrets in config files
• Supports MFA and SSO profiles
• Easy credential rotation

Keys

Uses explicit AWS access keys:

{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "keys",
        "info": {
            "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
            "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "region": "us-east-1"
        }
    }
}

Suited for CI/CD pipelines, containers, or cross-account access without AWS CLI.

Role

Assumes an IAM role via STS:

{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "role",
        "info": {
            "role_arn": "arn:aws:iam::123456789012:role/GluePublishRole",
            "external_id": "optional-external-id",
            "region": "us-east-1"
        }
    }
}

Uses temporary credentials. Ideal for cross-account publishing and fine-grained access control.

Optional Field: catalog_id

All three auth_mode values above also accept an optional catalog_id field in info, to target a Data Catalog in a different AWS account than the one whose credentials are used:

{
    "glue_catalog": {
        "ds_type": "aws_glue",
        "auth_mode": "profile",
        "info": {
            "profile": "my-aws-profile",
            "region": "us-east-1",
            "catalog_id": "123456789012"
        }
    }
}

If omitted, catalog_id defaults to the caller's own AWS account.

Understanding Glue Database Naming

Two ways to set the Glue database name

Unlike traditional databases, the AWS Glue Data Catalog has no server or database instance to connect to. What Glue calls a "database" is simply a namespace — a container for organizing tables within the catalog. The catalog itself (scoped by catalog_id, which defaults to your AWS account) is the top-level container.

By default, --publish_schema_pattern controls the Glue database name that will be created. For example, --publish_schema_pattern "datalake_{schema}" creates a Glue database called datalake_tpch_1.

--publish_database_name also exists and, when set, takes precedence over --publish_schema_pattern for Glue: it is used as a literal Glue database name (no token substitution).

Configuration Options

Option	Description	Default
--publish_target ID	Credential ID for Glue target (required)	-
--publish_schema_pattern PATTERN	Glue database naming pattern	EXT_{schema}
--publish_database_name NAME	Literal Glue database name; overrides --publish_schema_pattern when set	-
--publish_table_pattern PATTERN	Table naming pattern	{table}
--n_jobs N	Parallel workers for table creation	1

note

There is no --glue_skip_existing CLI flag. AWS Glue publishing always (re)creates tables; existing-table skip behavior is not exposed on the command line.

Dynamic Naming Patterns

Database and table names support token-based patterns.

Token restrictions

The {table} token can only be used in --publish_table_pattern, not in --publish_schema_pattern. Since the schema pattern controls the Glue database name, using {table} there would attempt to create a separate database per table, which is not supported. LakeXpress validates this before starting the export and will report an error.

Supported Tokens

Token	Description	Example Output
{schema}	Source schema name	tpch_1
{table}	Source table name	customer
{database}	Source database name	tpch
{date}	Current date (YYYYMMDD)	20251210
{timestamp}	Current timestamp (YYYYMMDD_HHMMSS)	20251210_143022
{uuid}	UUID4 (consistent per run)	a1b2c3d4-...
{subpath}	CLI --sub_path value	staging

Common Patterns

Date-Partitioned Databases

--publish_schema_pattern "lx_{schema}_{date}"
--publish_table_pattern "{table}"
--publish_target glue_catalog

# Results:
# Database: lx_tpch_1_20251210
# Tables: customer, orders, lineitem

Prefixed Databases

--publish_schema_pattern "datalake_{schema}"
--publish_table_pattern "{table}"
--publish_target glue_catalog

# Results:
# Database: datalake_tpch_1
# Tables: customer, orders, lineitem

Consolidated Multi-Schema

--source_schema_name schema1,schema2
--publish_schema_pattern "consolidated"
--publish_table_pattern "{schema}_{table}"
--publish_target glue_catalog

# Results:
# Database: consolidated
# Tables: schema1_customer, schema2_customer

Usage Examples

Ready-to-run commands for AWS Glue are in Examples & Recipes: AWS Glue.

See Also

• AWS Glue Reference - Data type mappings, CLI arguments, querying
• CLI Reference - All command-line options
• Examples & Recipes - Working command examples