Watermarks and windows¶

Navigate: Configure → Mappings → New Mapping or Edit Settings → Step 3: Watermark & CDC.

A watermark is how Paridity knows how far replication has got. It determines which rows fall inside each incremental window — and therefore what "complete" means for a given run. Get this wrong and you will chase differences that are just timing.

The Watermark & Windowing section starts collapsed. Click the title to expand it; its description lives inside.

Discovered candidates¶

When you pick a source table, Paridity inspects it and proposes watermark columns, shown as chips with a strategy and confidence. Clicking one applies it to both sides. The top candidate is applied automatically.

This means step 3 will usually not be empty when you arrive. Re-selecting the source table later will not overwrite a watermark you have already configured — auto-fill only runs while the watermark field is still empty.

Strategies¶

Choose per side. Prefer the highest option your systems support: the further down this list you go, the more you are inferring rather than observing.

Strategy	What it reads	Needs a field?
control_marker	A completion record your pipeline writes when a load finishes. Most reliable - eliminates guesswork. Target side only (see below).	no
delta_version	Databricks Delta table version / CDC offset.	no
commit_snapshot	The newest committed state of an Iceberg or Delta table (a strategy in the mapping form, Watermark & CDC step; Trino and Databricks systems and Delta and Iceberg file datasets; on Snowflake only the change token, source `snowflake_commit_time`, sequence domain). See Lakehouse tables and Delta Lake outside Databricks.	only for the `column` form
ingest_ts	A timestamp column set at load completion (not extraction start).	yes
partition_max	The maximum partition value (ADLS/Hive). Safe when partitions are atomic.	yes
oracle_scn	Oracle System Change Number.	no
lsn	Log Sequence Number (SQL Server / DB2).	no
timestamp	Max value of a source timestamp column. A fallback — it infers freshness from data rather than from the pipeline.	yes
file_manifest	A manifest file written when ingestion completes.	manifest path
snapshot	Full-table snapshot, no watermark.	no
partitioned_snapshot	Same as snapshot. Partitioning is set by the mapping's partition strategy, not by this type.	no
custom	An arbitrary SQL expression. Last resort — review at onboarding. Without a `field`, the value is table-level: run 1 and every later run compare the full dataset (advancement-gated), never a guessed column. Declare `field` to get an incremental window on that column.	SQL

control_marker is a target-side strategy. The marker table lives in the target system, and the source side has no marker settings, so saving a mapping with control_marker as the source watermark is refused (INVALID_WATERMARK_CONFIG). Put the marker on the target watermark and use a source-side strategy (for example timestamp or ingest_ts) on the source.

Do not infer freshness from business dates. A posting date or document date tells you about the business event, not about whether replication has delivered the row. For SAP and similar systems that lack a universal last-change timestamp, use replication metadata or an explicit snapshot mode.

Domain — the rule that catches people¶

Watermark domain must match the kind of value your strategy produces:

Domain	Use with
`time_utc`	`ingest_ts`, `timestamp`, `control_marker`, `partition_max`, `file_manifest`
`sequence`	`delta_version`, `lsn`, `commit_snapshot` with source `column` or `snowflake_commit_time`

Never mix domains within a mapping. An SCN and a timestamp are not comparable quantities.

Choosing delta_version, lsn, or commit_snapshot with source column or snowflake_commit_time while the domain is still time_utc is rejected on save with "watermark_domain must be 'sequence' when using a sequence-only watermark (target delta_version); got 'time_utc'" - the message names the offending side and type. The form does not switch the domain for you, so change it yourself when you pick a sequence strategy.

Pair a snowflake_commit_time side with a non-snapshot watermark on the other side (custom_sql SELECT 1 is the minimal form), or the token is never consulted: with snapshot on either side every run is a full compare and never ends no_new_data.

The rule is enforced on create and on edit, and on edit it is checked against the effective result — so changing the domain alone, while a stored watermark type still requires sequence, is rejected too.

oracle_scn and partition_max are deliberately not forced to sequence. An SCN is translatable to and from a timestamp, so oracle_scn is legitimate in either domain, and partition_max is a time_utc strategy. Only delta_version and lsn are sequence-only.

Windowing¶

Field	Values	Notes
Windowing mode	`fixed_interval` — a fixed time/sequence window; `watermark_range` — bounded by observed source-to-target advancement	The API also stores `partitioned_snapshot`; it has no runtime effect - partitioning comes from the partition strategy.
Window Interval (seconds)	minimum 60, default 3600	The minimum reconciliation window size.
Min Advance (seconds)	minimum 10, default 60, and must be less than the window interval	How far the watermark must move before a new window is worth opening. Prevents churn.

The form enforces these bounds directly, and the Min Advance box is capped at one second below whatever window interval you have entered — so an invalid pair is caught as you type rather than on save.

Readiness and CDC¶

The adjacent Readiness & CDC section decides when it is safe to start.

Field	Values	Notes
CDC Integration Mode	`advisory` — use the CDC window, fall back to querying; `strict` — halt if CDC is unavailable; `disabled` — ignore CDC entirely	Mapping-level only. A schedule or a single run cannot weaken it.
Max Latency Seconds (`max_lag_seconds`)	≥ 60	Maximum tolerated replication latency. A schedule may tighten this but never exceed it.
Quiescence Seconds	0, or 30–900	How long the target watermark must be stable before proof begins. The configured duration is waited in full.

Quiescence is waited in full, up to 900 seconds. The readiness phase heartbeats while it waits, so a long stability window does not delay detection of a genuinely stuck run.

900s is the ceiling the API accepts, because that is the longest the readiness phase will hold. If your pipeline delivers less often than that, quiescence is the wrong instrument — it infers stillness rather than observing completion. Use a control_marker watermark instead: your pipeline writes a record when a load finishes, and Paridity reads that signal directly.

With no replication profile attached, all three are disabled and CDC mode is forced to disabled — the form says so explicitly. Attach a replication profile to enable them.

With a custom replication tool, the controls are enabled but the platform cannot verify your pipeline's behaviour, so it warns that you must set latency and quiescence to match reality yourself.

Filters, SCD scope, and what a watermark sees¶

Two mapping settings restrict which rows are compared — they interact with watermarks differently, and the difference is deliberate:

Filter SQL (source/target) also scopes the watermark query. A static filter — a region, a company code, a partition — describes a fixed subset, and the watermark of that subset is the right freshness signal for it.
A declared SCD scope never scopes the watermark query. The current-rows boundary is mobile: rows leave it as part of normal operation. If the watermark only looked at current rows, a pure logical delete (a row closed with no new version written) would be invisible — the freshest change just left the scope, the watermark would not advance, and the run would report No new data while a real in-scope change went uncompared. The watermark therefore reads the whole table (through your Filter SQL, if any): the worst case of that choice is a comparison triggered by out-of-scope churn that finds nothing — compute cost, never missed detection.

Checking it worked¶

After a run, open the run detail and read the Data Window card: the mode, the window bounds, and the observed watermarks for each side. If you are ever unsure what a run actually compared, that card is the answer — and it is the first place to look when a result surprises you.