Logging and Monitoring
LakeXpress uses two logging systems: a database for structured export tracking and log files for operational details.
Database Logging
The export tracking database (SQL Server, PostgreSQL, or SQLite) records all export activity for reporting and troubleshooting.
Database Schema Overview
Key Tables
export_runs
Top-level record per export run:
- Unique run identifier
- Run type (full, incremental, custom)
- Source database and schema
- Storage configuration
- Start/completion timestamps and status
- Aggregate metrics (total, completed, failed jobs)
- Total rows exported and storage size
export_jobs
Per-table export job tracking:
- Job ID and associated run_id
- Step ID linking to export_steps
- Source table details (database, schema, table)
- Storage settings (type, path, format, compression)
- Parquet settings (row group size)
- FastBCP settings (distribution key, method, parallel degree)
- Results (file count, rows exported)
- Status (job_status, step_status)
- Retry count and timestamps (created, started, completed)
export_steps
Workflow step definitions:
- Step names and descriptions
- Links to jobs and log entries
export_files
Per-file tracking for generated Parquet files:
- File ID linked to parent job
- File name and full path
- Storage location (local, S3, Azure, GCS)
- File size, row count, status, creation timestamp
export_columns
Column metadata per exported table:
- Job ID + source column name (composite PK)
- Source database type
- Parquet column name (post-transformation)
- Source data type, precision, scale
- Ordinal position, length, nullability
export_log
Event log:
- Log ID and timestamp
- Links to run_id, job_id, step_id
- Log level (INFO, WARNING, ERROR, DEBUG)
- Event status, elapsed time (seconds)
- Source table context (database, schema, table)
- Event/error messages and error codes
partition_columns
FastBCP partitioning configuration:
- Partition config ID
- Source table identification (db type, database, schema, table)
- FastBCP distribution key, method, parallel degree
- Environment name, enable/disable flag
- Description and audit timestamps
File-Based Logging
Each run creates a log file named:
lx_YYYYMMDD_HHMMSS_<run_id>.log
Log Format
Structured format with timestamp, log level, process ID, and context:
[YYYY-MM-DD HH:MM:SS.fff+TZ :: LEVEL :: PID :: Context] Message
Sample Log Output
[2025-11-05 10:15:23.145+01:00 :: INFO :: 12345 :: MainProcess] Log file path: /var/log/lakexpress/lx_20251105_101523_a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d.log
[2025-11-05 10:15:23.146+01:00 :: INFO :: 12345 :: MainProcess] **** Starting export run ****
[2025-11-05 10:15:23.147+01:00 :: INFO :: 12345 :: MainProcess] Source: SQLServer - salesdb.dbo
[2025-11-05 10:15:23.148+01:00 :: INFO :: 12345 :: MainProcess] Target storage: S3 - s3://data-lake/exports/
[2025-11-05 10:15:23.250+01:00 :: INFO :: 12345 :: MainProcess] Discovered 15 tables for export
[2025-11-05 10:15:23.385+01:00 :: INFO :: 12345 :: Worker-1] Exporting table: orders (estimated 1.2M rows)
[2025-11-05 10:15:45.721+01:00 :: INFO :: 12345 :: Worker-1] Export completed: orders - 1,234,567 rows in 8 files (245 MB)
[2025-11-05 10:15:45.722+01:00 :: INFO :: 12345 :: Worker-1] Export elapsed time: 22.337 seconds
Log Levels
Set via --log_level (placed after the subcommand, e.g. sync --log_level DEBUG). Valid
choices are DEBUG, INFO, WARNING, ERROR (default INFO) -- CRITICAL is not an accepted
value and the code never emits a CRITICAL-level record:
- INFO: Progress, row counts, operational messages
- WARNING: Non-critical issues (schema differences, type conversions)
- ERROR: Failures preventing a table export
- DEBUG: SQL queries, file operations, diagnostics
Log Location
Defaults to the current working directory. Override with --log_dir:
./LakeXpress sync -a credentials.json --lxdb_auth_id lxdb_postgres --sync_id <SYNC_ID> --log_dir /var/log/lakexpress
Terminal Output
Logs also appear in the terminal, formatted with fixed (not severity-based) colors: the timestamp, level tag, process ID, and process name each render in their own consistent color regardless of whether the message is INFO, WARNING, or ERROR. In particular, the level tag is always cyan in standard mode (white in DEBUG mode) -- there is no green/yellow/red success/warning/error color-coding.