Troubleshooting

Missing Late-Arriving Rows

Symptom: Rows inserted in the source after a sync are never exported.

Cause: Each run exports up to MAX(column) read at run start and stores it as the watermark. A row committed later with a value at or below that watermark is outside every later run's range.

Solution: For datetime and date columns, set --incremental_safety_lag to the longest delay between a row's timestamp and its commit, so the most recent rows wait for the next run (see Safety Lag). Otherwise use an incremental column whose values only grow as rows are written (identity column, or a load timestamp set by the database), or reload the table periodically (see Resetting Watermarks).

Duplicate Rows After Resync

Symptom: Running the same sync twice produces duplicates.

Solutions:

1. Use the upsert strategy (recommended; not available on AWS Glue and Fabric Lakehouse, see Loading Strategies): configure the table with !upsert so existing rows are updated instead of added again. The source table needs a primary key:

--incremental_table "sales.orders:created_date:date!upsert"

2. Deduplicate in queries: If using append mode, use ROW_NUMBER() or QUALIFY at query time.

3. Use external tables: External tables over Parquet files replace files on each sync rather than inserting rows.

Watermark Not Advancing

Symptom: Watermark stays at the same value across syncs.

Solution: The watermark is MAX(column) read from the source at the start of each run, for every column type. If it does not change, no row with a higher value has arrived: check with the query below. If the source has newer values, check on the Watermarks page that last_exported_value in incremental_watermarks is updated (a sync failing before the end of the export does not update it).

SELECT MAX(created_at) FROM orders;
-- If this equals the stored watermark, there is no new data to export.

Performance Considerations

Column Selection

Choose the incremental column carefully:

-- GOOD: Indexed column, never updates
--incremental_table "sales.orders:order_id:int"

-- GOOD: Updated on insert only
--incremental_table "sales.orders:created_date:date"

-- CAUTION: May need resync if updated
--incremental_table "sales.orders:updated_at:datetime"

-- AVOID: May miss updates
--incremental_table "sales.orders:order_amount:int"

Indexing

Create an index on the watermark column:

CREATE INDEX idx_orders_created_at ON orders(created_at);

Combining Incremental and Full Exports

--incremental_table "fact.transactions:txn_date:date"
--incremental_safety_lag 3600
--n_jobs 4
--fastbcp_p 8
--fastbcp_table_config "fact.transactions:Ntile:txn_id:8"

• Fact tables configured with --incremental_table sync incrementally
• All other tables in the schema are fully exported (useful for dimension tables)
• --fastbcp_p alone does not parallelize an incremental table on PostgreSQL, Oracle or SQL Server: the automatic methods (Ctid, Rowid, Physloc) cannot be used with the incremental query and are replaced by None. The --fastbcp_table_config entry sets a method with a key column (Ntile on txn_id) for it. See Incremental Tables.

Multiple Watermark Columns

Track changes in multiple columns with separate configurations:

# Configuration 1: Track by create date
lakexpress config create \
  ... \
  --incremental_table "sales.transactions:created_at:datetime"

# Configuration 2: Track by update date
lakexpress config create \
  ... \
  --incremental_table "sales.transactions:updated_at:datetime"

Run both syncs to capture creates and updates separately.

See Also

• Incremental Sync OverviewConfiguration syntax and supported column types
• Loading StrategiesAppend vs upsert strategies
• WatermarksWatermark tracking, safety lag, and resetting