Logging and Monitoring

LakeXpress uses two logging systems: a database for structured export tracking and log files for operational details.

Database Logging

The export tracking database (PostgreSQL, SQL Server, MySQL, SQLite or DuckDB) records all export activity for reporting and troubleshooting.

Database Schema Overview

Key Tables

Tables grouped by role. Click a table in the schema above to jump to its description, and Show in schema on a card to go back up.

Configuration

export_syncs

One row per sync configuration, created by config create and read by every sync run.

• sync_id, environment name
• Source credentials, database and schemas
• Include and exclude patterns, min and max rows
• Target storage, output directory, sub-path
• Publish target, method, database, schema and table patterns
• FastBCP path, parallelism, number of jobs
• Compression, large-table threshold
• Incremental safety lag
• Manifest, error action, enabled flag
• LakeXpress version, audit timestamps

export_steps

Workflow step definitions, referenced by jobs, publish jobs and log entries.

• step_id
• Step name and description

partition_columnsnot in the schema

FastBCP partitioning configuration per source table.

• Partition config ID
• Source table (db type, database, schema, table)
• FastBCP distribution key, method, parallel degree
• Environment name, enabled flag
• Description, audit timestamps

Export runs and jobs

export_runs

Top-level record of each export run of a sync.

• run_id, sync_id, run sequence
• Run type: incremental when the sync has incremental tables, full otherwise
• Source database and schemas
• Storage configuration
• Start and completion times, status
• Total, completed and failed jobs
• Rows exported, storage size
• LakeXpress and FastBCP versions

export_jobs

One row per table exported in a run.

• job_id, run_id, step_id
• Source db type, database, schema, table, estimated rows
• Storage type, base path, file format, compression
• Parquet row group size
• FastBCP distribution key, method, parallel degree
• Files and rows exported
• Job status, step status, retry count
• Created, started and completed times

Files and columns

export_files

One row per Parquet file written by a job.

• file_id, job_id
• File name and full path
• Storage location (local, S3, Azure, GCS)
• File size, row count, status
• Creation time

export_columns

Column metadata of each exported table. Primary key: job_id and source column name.

• job_id, source column name
• Source db type and data type
• Parquet column name (after transformation)
• Ordinal position, length, precision, scale
• Nullability, primary-key flag

Logs

export_log

Event log of runs, jobs and steps.

• log_id, timestamp
• run_id, job_id, step_id
• Log level (INFO, WARNING, ERROR, DEBUG)
• Event status, elapsed time in seconds
• Source database, schema and table
• Event message, error code and message

Incremental sync

incremental_tables

One row per table synced incrementally in a sync, from its --incremental_table spec.

• incremental_table_id, sync_id
• Source schema and table
• Incremental column and type (date, datetime, int, bigint, auto)
• Lower bound inclusive or exclusive, start value
• Loading strategy (append, upsert, delete_insert)
• Mode, publish method, primary key columns
• Enabled flag, description, audit timestamps

incremental_watermarks

Current watermark of each incremental table: where the next sync starts.

• watermark_id, incremental_table_id
• Last exported value, d_new
• Schema version and hash
• Active run and its start time
• Last export time, last run, rows exported
• Version, update time

incremental_export_history

One row per incremental export of a table, with the window it covered.

• history_id, watermark_id, run_id
• Window bounds d_old and d_new
• Export mode, rows exported, status
• Output path
• Start and completion times
• Replay flag, error message

Publishing

publish_runs

One row per publish run, linked to the export run it publishes.

• publish_run_id, export_run_id
• Target platform, credentials, database, stage
• Platform configuration, publish options
• Status, start and completion times
• Total, completed and failed jobs
• Schemas, tables and views created or failed
• Rows loaded, error message

publish_jobs

One row per table published in a publish run.

• publish_job_id, publish_run_id, export_job_id, step_id
• Source schema and table
• Target platform, database, schema, table
• Table type, creation and status
• View name, creation and status
• Data loaded, rows loaded, load status
• Job status, retry count, max retries
• Error code and message
• Created, started and completed times, elapsed seconds

File-Based Logging

Each run creates a log file named: lx_YYYYMMDD_HHMMSS_<run_id>.log

Log Format

Structured format with timestamp, log level, process ID, and context:

[YYYY-MM-DD HH:MM:SS.fff+TZ :: LEVEL :: PID :: Context] Message

Sample Log Output

[2025-11-05 10:15:23.145+01:00 :: INFO :: 12345 :: MainProcess] Log file path: /var/log/lakexpress/lx_20251105_101523_a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d.log
[2025-11-05 10:15:23.146+01:00 :: INFO :: 12345 :: MainProcess] **** Starting export run ****
[2025-11-05 10:15:23.147+01:00 :: INFO :: 12345 :: MainProcess] Source: SQLServer - salesdb.dbo
[2025-11-05 10:15:23.148+01:00 :: INFO :: 12345 :: MainProcess] Target storage: S3 - s3://data-lake/exports/
[2025-11-05 10:15:23.250+01:00 :: INFO :: 12345 :: MainProcess] Discovered 15 tables for export
[2025-11-05 10:15:23.385+01:00 :: INFO :: 12345 :: Worker-1] Exporting table: orders (estimated 1.2M rows)
[2025-11-05 10:15:45.721+01:00 :: INFO :: 12345 :: Worker-1] Export completed: orders - 1,234,567 rows in 8 files (245 MB)
[2025-11-05 10:15:45.722+01:00 :: INFO :: 12345 :: Worker-1] Export elapsed time: 22.337 seconds

Log Levels

Set via --log_level (placed after the subcommand, e.g. sync --log_level DEBUG). Valid choices are DEBUG, INFO, WARNING, ERROR (default INFO) -- CRITICAL is not an accepted value and the code never emits a CRITICAL-level record:

INFOdefault

Progress, row counts, operational messages

WARNING

Non-critical issues (schema differences, type conversions)

ERROR

Failures preventing a table export

DEBUG

SQL queries, file operations, diagnostics

Log Location

Log files are written one per run, named lx_<date>_<time>_<run_id>.log. An export run also saves a copy of its configuration next to it (lx_<date>_<time>_<run_id>.yml); sync[publish] writes its own log file, named after the export run it publishes. They go to the current working directory by default. Override with --log_dir (the directory is created if needed):

./LakeXpress sync -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID> --log_dir /var/log/lakexpress

Terminal Output

Logs also appear in the terminal, formatted with fixed (not severity-based) colors: the timestamp, level tag, process ID, and process name each render in their own consistent color regardless of whether the message is INFO, WARNING, or ERROR. In particular, the level tag is always cyan in standard mode (white in DEBUG mode) -- there is no green/yellow/red success/warning/error color-coding.