Database Connections

LakeXpress uses two distinct database roles, each requiring its own connection entry in the credentials file:

• LakeXpress DB — An internal database used by LakeXpress to store configuration, track sync state, and log pipeline runs. This is not a source of data; it is LakeXpress's own bookkeeping store. You point LakeXpress at it with --lxdb_auth_id. It is set up once, at installation: see Set up the LakeXpress DB.
• Source Database — The database you want to extract data from (e.g., a production PostgreSQL or Oracle instance). You point LakeXpress at it with --source_db_auth_id.

Both connections are defined in the same credentials JSON file, distinguished by the key name you give each entry.

Supported Databases

Source Databases

LakeXpress reads tables from these databases. Each page covers the connection settings and the privileges the export user needs.

• PostgreSQLCtid-based parallel export
• Oraclethin and thick modes, Rowid-based parallel export
• SQL ServerPhysloc-based parallel export
• MySQLpartition detection
• MariaDBdedicated mariadb ds_type
• Teradatarequires the teradatasql driver
• SAP HANArequires hdbcli, and HDBDOTNETCORE for FastBCP

ADBC Extraction

FastBCP extracts SQL Server, PostgreSQL and Teradata through the arpe.io ADBC drivers it embeds, by default. Set "use_adbc": false on a source, at the same level as ds_type, to use FastBCP's native driver instead. Oracle has an ADBC driver too, used only with "use_adbc": true (see Oracle).

Source	Default extraction	Other choice
SQL Server	adbc_mssql (ADBC)	mssql (native), with "use_adbc": false
PostgreSQL	adbc_pgsql (ADBC)	pgcopy (native), with "use_adbc": false
Oracle	oraodp (native)	adbc_oracle (ADBC), with "use_adbc": true
Teradata	adbc_teradata (ADBC)	teradata (native), with "use_adbc": false
MySQL, MariaDB	mysql (native)	none
SAP HANA	hana (native)	none

• ADBC only changes the extraction to Parquet. LakeXpress still discovers schemas, tables and columns with its own Python drivers.
• MySQL, MariaDB and SAP HANA have no ADBC driver. "use_adbc": true on them logs a warning and uses the native driver.
• use_adbc must be true or false; any other value stops the sync before the export with exit code 10.
• The ADBC drivers need the ADBC block of the license file (exit code 13 without it), and ship for linux-x64 and win-x64.
• ADBC writes timezone-aware timestamps as UTC and keeps microseconds. See the ADBC section of each source page for the type differences with the native drivers.

LakeXpress DB

The database where LakeXpress keeps its configurations and run history. How to create it and choose a backend: Set up the LakeXpress DB.

• PostgreSQL
• SQL Server
• MySQL
• SQLite
• DuckDB

Credentials File Format

All database credentials are stored in a single JSON file, credentials.json in this documentation (any name works: pass it with -a). Each top-level key is a connection identifier you choose — you reference these identifiers on the command line via --lxdb_auth_id and --source_db_auth_id. The file can hold as many connection entries as you need.

Create a JSON file with your database credentials:

{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_oracle": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-server.com",
      "port": 1521,
      "database": "orclpdb1",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  },
  "lxdb_sqlite": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/path/to/lakexpress_log.sqlite"
    }
  }
}

Every command takes the file with -a and picks entries by name: --lxdb_auth_id for the LakeXpress DB, --source_db_auth_id for the source database. For complete commands, see the Getting Started guide and Examples & Recipes.

Publishing Credentials

Publishing targets use their own credential entries in the same credentials.json file. See each publishing guide for authentication setup:

• Snowflake
• Databricks
• Microsoft Fabric
• BigQuery
• MotherDuck
• AWS Glue
• Amazon Redshift
• DuckLake

Security Best Practices

Credential storage

• Never commit credentials to version control
• Restrict credentials.json permissions:

  chmod 600 credentials.json

• Use a separate credentials file per environment (dev, staging, prod)

Least privilege

• Create dedicated database users for LakeXpress
• Grant only SELECT on source databases: LakeXpress never writes to them
• Grant full privileges on the LakeXpress DB: LakeXpress creates and updates its own tracking tables there

Network security

• Use SSL/TLS connections
• Restrict database access by IP
• Use VPN or private networks

Password management

• Use strong passwords (12+ characters, mixed case, numbers, symbols)
• Rotate passwords regularly
• Consider secret managers (AWS Secrets Manager, HashiCorp Vault), or the environment variables below

Environment Variables for Credentials

LakeXpress supports $env{VAR_NAME} syntax in any string value in the credentials JSON. An error is raised if a referenced variable is not set.

{
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  }
}

Set variables before running LakeXpress:

Windows (PowerShell)

$env:LX_PG_USER = "dataexport"
$env:LX_PG_PASSWORD = "SecureP@ssw0rd"
$env:LX_MSSQL_USER = "sa"
$env:LX_MSSQL_PASSWORD = "StrongP@ssw0rd"

Windows (cmd)

set LX_PG_USER=dataexport
set LX_PG_PASSWORD=SecureP@ssw0rd
set LX_MSSQL_USER=sa
set LX_MSSQL_PASSWORD=StrongP@ssw0rd

Linux

export LX_PG_USER="dataexport"
export LX_PG_PASSWORD="SecureP@ssw0rd"
export LX_MSSQL_USER="sa"
export LX_MSSQL_PASSWORD="StrongP@ssw0rd"

Plain-text values still work -- values without $env{ are returned as-is.

See Also

• Quick Start GuideGetting started
• CLI ReferenceCommand-line options
• Intermediate StorageStorage configuration