Mappings¶
Navigate: Configure → Mappings (/mappings) → New Mapping, or open
a mapping and choose Edit Settings.
A mapping is one source object compared against one target object, plus the policy that governs the comparison. It is the central object in Paridity.
The four steps¶
| Step | Covers |
|---|---|
| 1. Source & Target | Systems, objects, key columns, replication profile |
| 2. Column Mapping | Source↔target column pairing (creation only) |
| 3. Watermark & CDC | See Watermarks and windows |
| 4. Advanced | Filters, compute, diff limits, evidence, drift, ownership, partitioning |
Step 2 is hidden when editing, so the sidebar reads 1, 3, 4. Column mappings for an existing mapping are edited in the dedicated column-mapping editor.
Step 1 — Source & Target¶
| Field | Notes |
|---|---|
| Name | Required, unique per tenant. |
| Source / Target System | Fixed after creation. |
| Source / Target Table | Type a reference or pick from discovery. File-based systems use a path (bucket/path/file.parquet); everything else uses SCHEMA.TABLE. Fixed after creation — shown read-only when editing. |
| Key Columns | Comma-separated. Creation only. Auto-filled from the detected primary key — the label turns green to say so, and clears the moment you type. |
| Replication Profile | Optional but strongly recommended. Only profiles whose route matches your two systems are listed. Changeable after creation — see below. |
Why the tables and key columns are fixed. They define what the mapping is and what its comparison covers. A mapping keeps its identity, its history and its open mismatch keys across an edit, so repointing it at a different table would leave every earlier run's evidence describing a dataset the mapping no longer reads. To change them, create a new mapping. To pick up added or changed columns on the same tables, use Refresh Schema on the mapping detail page. Refresh Schema only applies a schema it could actually read from both systems: if either side cannot be reached, the table is missing, or the connection lacks catalog permissions, the refresh is refused with a message naming the side and the reason, and your field configuration is left exactly as it was. Fix the connection or object reference and refresh again.
Changing the replication profile. You can move a mapping to a different profile, or clear the profile entirely, from this step — and the change now takes effect (before #2217 the form reported success and the profile never moved). Only profiles connecting the same two systems are accepted. If your watermark settings are still exactly the ones the old profile's template supplied, they follow the new profile; anything you changed yourself is left alone. Clearing the profile is refused while latency or quiescence controls are still set — clear them in the same save.
Selecting a source table pre-populates step 3. It sets the watermark strategy and field on both sides, and for file-based sources it also guesses a manifest path. Once a watermark field is set, changing the table again leaves your configuration alone — auto-fill only runs while the field is empty. Check step 3 before saving regardless: the guess is a starting point, not a decision.
The object picker tells you whether discovery succeeded — "N objects discovered" — or that the connector was unreachable and you should type the reference manually. An unreachable connector does not block mapping creation.
Step 4 — Advanced¶
Filters¶
Source / Target filter SQL are WHERE-clause fragments applied before
comparison. Omit the WHERE keyword itself.
They are deliberately restricted: no statement terminators, no comments, no DDL/DML keywords, 2000 characters maximum. The form shows a soft warning for suspicious patterns; the server enforces the rules and rejects violations with a specific reason.
SCD2 / bitemporal tables — declared scope¶
An SCD2 (slowly-changing-dimension) or bitemporal table legitimately keeps closed history rows the other side no longer shows. Compared whole, such a pair fails forever on data that is actually correct. Declare SCD scope (compare current rows only) — in the mapping editor's Advanced step — scopes the comparison to the current rows of the side that holds history:
- Scoped side — which side (source or target) holds the history.
- How current rows are marked — one of three typed markers: a flag column
equal to a value (
current_flag = 'Y'), an end-date column equal to a far-future sentinel (valid_to = 9999-12-31), or an end-date column that isNULLwhile the row is current. You pick columns and values — you never write SQL, and the platform never infers any of this: an undeclared scope is a false-pass machine, so the scope is always an explicit, versioned, checksummed declaration. - Validity interval columns (optional) —
valid_from/valid_to, required if you opt into the interval-overlap check below.
The SCD bookkeeping columns (flag, validity dates, version numbers) are simply not mapped as comparison fields — the compare covers the business columns you map, and the scope decides which physical rows carry them. A scope column cannot also be a mapped comparison field; the editor blocks that combination.
Every run under a declared scope also checks the scope's own premise (invariants, evaluated on the scoped side before the proof):
- Exactly one current row per business key — always on; a scope with duplicate current rows has no honest row identity.
- No overlapping validity intervals (opt-in, needs the interval columns; costs an extra self-join per run) — catches history corruption even when the current rows still match.
A violated invariant never silently passes or fails the run: the run keeps its data verdict but is reclassified SCD invariant violated, excluded from pass counts, and fires a warning notification — see Runs.
Runs under a declared scope carry a “Scoped: SCD current rows” badge on the run detail and evidence pages: a scoped pass proves the current rows, not the whole table, and the display never hides that. Scope declarations are supported on live-verified engines only (currently Postgres and Snowflake); declaring a scope on any other engine is rejected outright rather than applied on faith.
One deliberate asymmetry: the scope applies to the comparison, never to watermark queries — see Watermarks for why.
Compute¶
| Field | Range | Notes |
|---|---|---|
| Compute Mode | pushdown_only, hybrid, in_memory_only |
Where SQL executes. hybrid falls back to the reference implementation for unsupported operations; in_memory_only is the most portable and the slowest. |
| Max Concurrency | 1–50 | Capped by each system's own ceiling. |
| Max Diff Rows | ≥ 1000 | Cap on rows handed to the diff engine. |
| Query Timeout | 10–3600s | Overrides both systems for this mapping. Size it for the proof query at full volume. |
| Max Partition Concurrency | 1–20 | Parallel partition workers. |
Diff safety limits¶
Caps on how much diff evidence one run may produce. Each resolves as the minimum across platform, tenant, mapping and schedule — a run override can lower them but never raise them.
Egress Row Limit (rows pulled back for diffing — the main egress control) ·
Per-Column Diff Limit · Global Diff Cell Limit · Fail-Fast Threshold
(%) (skip the diff phase entirely when this share of rows mismatch — the run
is already conclusive) · Diff Truncation Method (First N, Random sample,
Stratified; schedules and runs may only choose the first two).
Evidence¶
Coverage mode — full field diff, limit to N rows, or a percentage. Sampled
diffs are labelled is_sampled = true, and count and hash proof stay
exhaustive regardless: sampling reduces the evidence you keep, never the
rigour of the pass/fail decision.
Store Raw Diff Values is an AND-chain across tenant, mapping and field — any
FALSE suppresses storage and sets values_masked. Retention for
summaries, mismatch keys and field diffs cannot go below the tenant minimums.
Schema drift¶
Fail on Schema Drift, plus a Lost Reconciled Column policy (halt, or
continue_excluding to drop the column and keep going) and separate handling
for additive and compatible drift (warn / halt / auto-accept).
Losing a key column always halts, whatever the policy says. Without the key there is no row identity to reconcile on, so the run stops rather than comparing what is left.
Other breaking changes — type narrowing, NOT NULL added, a collation change —
are classified as breaking and then follow the policy. Under
continue_excluding the affected column is dropped from the comparison and the
run is reported as Schema drift — Available data passed: it is counted as an
issue, never as a clean pass, so reduced coverage is always visible.
Unverifiable Schema policy (schema_unverified_policy; API-configurable,
UI pending): what happens when the platform cannot verify a side's schema at
run time — introspection failed (permission loss, catalog error) or the engine
cannot introspect at all.
proceed(default): the data is still compared — but the run displays Pass · Schema unverified (amber; or the calmer Schema not verifiable for engines declared non-introspecting), records which side(s) were unverified and why, and is excluded from pass counts.halt: the run stops (Schema unverified · Not compared) rather than comparing against a schema that could not be verified — for high-assurance mappings.
The policy only controls the halt. The labeling, the evidence record, and the pass-count exclusion are absolute — no setting renders an unverified schema as verified. A tenant-level default exists in the database only (no API/UI yet), like the tenant term of the drift policy.
Ownership, close mode, partitioning¶
Data steward / technical owner / business owner / DBA group and business
criticality (Regulatory, Business Critical, Operational, Dev / Test).
Close Mode Eligible and Tags control inclusion in bulk close-mode
toggles. Auto-ticket opens a ServiceNow or Jira ticket on terminal failure,
with a deep link to the signed evidence bundle. Switching auto-ticket off also
un-assigns the ticket channel, and a saved edit that clears the channel now
takes effect (before #2208 the form reported success and the channel stayed
assigned). A mapping cannot auto-ticket to no channel: leaving the toggle on
with no channel selected is rejected on save. You pick the ticket channel by
name from your enabled ServiceNow and Jira channels. If none exist yet, the form
says so and links to creating one. Disabled channels are listed by name so you can
see why they are missing from the choices. A saved channel that has since been
disabled or deleted is flagged, because no ticket opens until you enable it or
pick another.
Partition Strategy splits a large window into independently proven and
diffed chunks: Time (with a grain and lookback), Key Range, List, or
Composite (a two-dimensional grid). A live estimate warns when the count is
large; above 256 it suggests a coarser granularity. Per-partition SQL scoping
is decided per side and per (engine, partition kind, column-type
family) cell — see the live-verified matrix in the
configuration reference.
On a cell that is not live-verified the partition is still proven and diffed
correctly — coverage is unaffected — but its proof runs over the whole window
on that side rather than its own slice, so the speed benefit you configured is
not realised there. The mapping detail page shows the pre-run prediction per
component and side (a time partition under a non-UTC timezone reads
"classified at run time" until a run records its frame), and the run page
shows what each run actually recorded, partition by partition, with the
reason — rather than leaving you to discover it from timings. How much faster
a partitioned proof actually is depends on your engines' capacity to run the
slices concurrently: each slice's proof is about N times cheaper on its own,
but slices that run at the same time compete for the same engine — the
measured numbers (a 50,000-row Postgres → Trino mapping, four slices, serial
vs concurrent) are in the platform's benchmark record
(docs/performance/benchmarks.md, § #2283); nothing here promises a speed-up.
Test mapping tags all runs as TEST so they can be bulk-deleted.
Retiring a mapping¶
A mapping that has run history cannot be deleted. Those runs, and the configuration snapshots attached to them, are the audit record proving what was compared and under which policy — so removing the mapping would remove the evidence. To retire one, disable it: it stops scheduling and running, its history stays intact and searchable, and the evidence ages out under your retention policy.
A mapping that has never run can be deleted outright. If it belongs to a
mapping group, the delete asks you to remove it from the group first, or accept
that with force.
Deleting a replication profile that mappings still use is refused for a
different reason: it is allowed, but only deliberately. Detaching a mapping
from its profile is not a neutral act — readiness short-circuits and CDC
integration is forced to disabled, so the mapping stops waiting for
replication to finish delivering the window. Re-point the mappings first, or
confirm the detach explicitly.
Validation you may hit¶
Most rejections are explicit. The ones worth anticipating:
| Situation | What happens |
|---|---|
| Business Domain cleared | Rejected — it is required, though the field carries no asterisk on step 4. |
delta_version/lsn with domain time_utc |
Rejected. Set the domain to sequence. |
| Min Advance ≥ Window Interval | Rejected. |
| Latency or quiescence set with no replication profile | Rejected - those controls need a profile. Clearing the profile while they are still set is rejected for the same reason; clear both in one save. |
| A replication profile connecting different systems | Rejected - a profile's latency and quiescence signals describe its own two systems, so attaching a foreign one would gate this mapping's runs on another pipeline. |
| A partition strategy that yields zero partitions | Rejected — a partitioned run over zero partitions compares nothing. |
| Overlapping or duplicate enumerated partitions | Rejected, with the specific violation named — duplicate list values, duplicate range ids, overlapping [lo, hi) intervals, or duplicate time slices. An overlapping partition double-reports its rows' evidence, because every containing partition proves and diffs them. |
| Tenant at its active-mapping limit | Rejected with an upgrade message. At 80% you get a warning but the mapping is created. |
| Someone else edited the mapping meanwhile | Rejected — reload and re-apply, so you never silently overwrite. |
Some numeric inputs allow values the platform rejects (window interval, min advance, max diff rows). Use the ranges in the tables above.
Two editors, one of them stale¶
The mapping detail page has an Edit settings dialog that is not the same as the full-page editor: it exposes fewer fields, uses different labels, and its business criticality options are not values the API accepts.
Use the full-page editor (New Mapping / Edit Settings →
/mappings/<id>/edit) — it is the maintained one, and it is what this guide
describes.