Version: 0.4 (Latest)

Database Connections

LakeXpress uses two distinct database roles, each requiring its own connection entry in the credentials file:

LakeXpress DB — An internal database used by LakeXpress to store configuration, track sync state, and log pipeline runs. This is not a source of data; it is LakeXpress's own bookkeeping store. You point LakeXpress at it with --lxdb_auth_id. It is set up once, at installation: see Set up the LakeXpress DB.
Source Database — The database you want to extract data from (e.g., a production PostgreSQL or Oracle instance). You point LakeXpress at it with --source_db_auth_id.

Both connections are defined in the same credentials JSON file, distinguished by the key name you give each entry.

Supported Databases

Source Databases

Database	Support Status	Notes
PostgreSQL	Supported	Ctid-based parallel export
Oracle	Supported	Thin and thick modes, Rowid-based parallel export
SQL Server	Supported	Physloc-based parallel export
MySQL	Supported	Partition detection
MariaDB	Supported	Dedicated `mariadb` ds_type
Teradata	Supported	Requires `teradatasql` driver
SAP HANA	Supported	Requires `hdbcli` and HDBDOTNETCORE for FastBCP

LakeXpress DB

PostgreSQL, SQL Server, MySQL, SQLite and DuckDB. How to create it and choose a backend: Set up the LakeXpress DB.

Credentials File Format

All database credentials are stored in a single JSON file, credentials.json in this documentation (any name works: pass it with -a). Each top-level key is a connection identifier you choose — you reference these identifiers on the command line via --lxdb_auth_id and --source_db_auth_id. The file can hold as many connection entries as you need.

Create a JSON file with your database credentials:

{
  "lxdb_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_LXDB_USER}",
      "password": "$env{LX_LXDB_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "lakexpress_log"
    }
  },
  "source_oracle": {
    "ds_type": "oracle",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_ORACLE_USER}",
      "password": "$env{LX_ORACLE_PASSWORD}",
      "server": "oracle-server.com",
      "port": 1521,
      "database": "orclpdb1",
      "lib_dir": "/opt/oracle/instantclient_19_8"
    }
  },
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  },
  "lxdb_sqlite": {
    "ds_type": "sqlite",
    "auth_mode": "filesystem",
    "info": {
      "filepath": "/path/to/lakexpress_log.sqlite"
    }
  }
}

Every command takes the file with -a and picks entries by name: --lxdb_auth_id for the LakeXpress DB, --source_db_auth_id for the source database. For complete commands, see the Getting Started guide and Examples & Recipes.

Publishing Credentials

Publishing targets use their own credential entries in the same credentials.json file. See each publishing guide for authentication setup:

Security Best Practices

1. Credential Storage

Never commit credentials to version control
Restrict credentials.json permissions:
```
chmod 600 credentials.json
```
Use a separate credentials file per environment (dev, staging, prod)

2. Least Privilege

Create dedicated database users for LakeXpress
Grant only SELECT on source databases — LakeXpress never writes to them
Grant full privileges on the LakeXpress DB — LakeXpress needs to create and update its own tracking tables there

3. Network Security

Use SSL/TLS connections
Restrict database access by IP
Use VPN or private networks

4. Password Management

Use strong passwords (12+ characters, mixed case, numbers, symbols)
Rotate passwords regularly
Consider secret managers (AWS Secrets Manager, HashiCorp Vault)

5. Environment Variables for Credentials

LakeXpress supports $env{VAR_NAME} syntax in any string value in the credentials JSON. An error is raised if a referenced variable is not set.

{
  "source_postgres": {
    "ds_type": "postgres",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_PG_USER}",
      "password": "$env{LX_PG_PASSWORD}",
      "server": "localhost",
      "port": 5432,
      "database": "production_db"
    }
  },
  "source_mssql": {
    "ds_type": "mssql",
    "auth_mode": "classic",
    "info": {
      "username": "$env{LX_MSSQL_USER}",
      "password": "$env{LX_MSSQL_PASSWORD}",
      "server": "mssql-server.com",
      "port": 1433,
      "database": "sales_db"
    }
  }
}

Set variables before running LakeXpress:

Windows (PowerShell)
Windows (cmd)
Linux

$env:LX_PG_USER = "dataexport"
$env:LX_PG_PASSWORD = "SecureP@ssw0rd"
$env:LX_MSSQL_USER = "sa"
$env:LX_MSSQL_PASSWORD = "StrongP@ssw0rd"

set LX_PG_USER=dataexport
set LX_PG_PASSWORD=SecureP@ssw0rd
set LX_MSSQL_USER=sa
set LX_MSSQL_PASSWORD=StrongP@ssw0rd

export LX_PG_USER="dataexport"
export LX_PG_PASSWORD="SecureP@ssw0rd"
export LX_MSSQL_USER="sa"
export LX_MSSQL_PASSWORD="StrongP@ssw0rd"

Plain-text values still work -- values without $env{ are returned as-is.

Supported Databases​

Source Databases​

LakeXpress DB​

Credentials File Format​

Publishing Credentials​

Security Best Practices​

1. Credential Storage​

2. Least Privilege​

3. Network Security​

4. Password Management​

5. Environment Variables for Credentials​

See Also​