How Paridity guarantees accuracy - and how to verify it yourself¶

The claim¶

For every engine pair marked Verified in the table below, Paridity detects every byte-level difference between a source and target dataset - with zero known false passes - using a deterministic cross-engine canonical hash comparison, not sampling and not inference. A false pass (reporting a dataset clean when it is not) is treated as a compliance failure, never an acceptable trade-off.

This is a narrow, falsifiable claim. It does not claim every supported engine is at the same verification depth (see Limitations), and it does not claim Paridity infers or judges correctness - correctness is always computed, never guessed. It is also scoped to a stated threat model - the proof defends against replication faults and corruption, not against a privileged insider deliberately tampering with the databases being compared (see Threat model); stating that scope precisely is part of the claim.

How it works (deterministic)¶

Aligned time window. Before any comparison, Paridity aligns the source and target to the same point in time using watermark state (CDC offset, control marker, or snapshot) - this is what makes it CDC-aware: comparing datasets mid-replication produces false positives, so Paridity waits for a stable, comparable window rather than comparing whatever happens to be in each table right now. Where the storage engine itself can change the picture between the readiness instant and the read, the read is pinned or merge-resolved rather than taken at "latest": a Snowflake native table with Time Travel retention is read AT(TIMESTAMP) of the readiness instant (views opt in per mapping with read_pin_views; a table with retention 0 is read unpinned and logged as such, never silently), and a ClickHouse ReplacingMergeTree side is read through FINAL when the mapping declares cdc_table_semantics (replacing_merge_tree_final or clickpipes_peerdb) - an operator declaration, never an inference.
Proof phase - hash_total. Paridity computes a hash_total - an aggregate fingerprint of every row in the window - independently on the source and the target, then compares the two values. This is an O(1) pass/fail signal: it proves an entire dataset identical (or not) without pulling the rows across the network. hash_total is not a checksum in the informal sense; it is two independent commutative aggregates (an XOR and a SUM of per-row hash integers) so that accidental cancellation between differing rows is statistically negligible - quantified, and scoped precisely, in Threat model below.
Canonicalization - the part that makes cross-engine comparison possible. Every column value is normalized to an identical string form before it is hashed, regardless of which database produced it:
NULL → a literal null token (<NULL> by default), and empty strings are folded into it only if configured to.
Numeric values → fixed-point decimal strings at a configured scale (e.g. 1250.5 at scale 4 → "1250.5000") - this is what makes an Oracle NUMBER and a Snowflake NUMBER hash identically for the same value.
Timestamps → normalized to UTC and rendered as ISO 8601 (2024-03-15T14:30:00.123Z) at a configured precision, never a database-local format.
Row fields are joined with the ASCII Unit Separator (\x1F) as the delimiter - chosen specifically because it cannot appear in normal text, and any literal occurrence is escaped by doubling it.
Column order is fixed per mapping (hash_column_order) and stored in the resolved config snapshot, so it is identical on both sides for the life of the run.

Full canonicalization rules are available to reviewers on request through your Paridity account contact. 4. row_hash - key extraction. If hash_total does not match, Paridity computes a SHA-256 row_hash per row on both sides and correlates them to identify exactly which rows differ - without re-comparing rows that already match. 5. field_diff - the deliverable. For each mismatched row, Paridity compares column-by-column and records the specific field(s) that differ, with the source and target values (subject to masking policy). This is the evidence an operator or auditor actually reads.

Note: AI never determines correctness in Paridity. Correctness is always established deterministically - by aligned time windows, hash_total aggregate fingerprints, row_hash per-row fingerprints, and field_diff column comparisons. AI assists triage and explanation of already-computed results only; it never decides whether data matches and never auto-confirms an operator action.

Threat model¶

The proof phase is an integrity check for replication correctness, and the accuracy claim above is scoped to that threat model. Paridity states this explicitly rather than leaving it implied.

What the proof defends against. Non-adversarial divergence between source and target: replication defects (missed, duplicated, or partial applies), corruption, canonicalization and schema drift, in-flight replication latency. hash_total is a homomorphic multiset hash - two commutative aggregates (XOR and SUM) over per-row SHA-256 fingerprints. Faults do not choose their bytes, so their contribution to the aggregate is effectively random, and the probability that a genuinely different dataset produces a matching aggregate is ≈10⁻¹⁹ per comparison (quantified below; smaller still on engines where the XOR component is real). Row counts are compared alongside the aggregate, so any fault that adds or removes rows is caught by count alone.

What the proof does not defend against. A malicious insider who can write the datasets being compared and knows the construction. Any 64-bit SQL-computable aggregate - however strong the per-row hash - can in principle be defeated by an adversary who modifies a row and then crafts other rows in the same window whose fingerprint deltas cancel in the aggregate (a generalized-birthday search, computationally feasible at this width). This is a property of every SQL-pushable aggregate construction, not an implementation defect. Adversarially secure alternatives (LtHash-class multiset hashes) cannot be pushed down into your engines as SQL - adopting one would trade away the property that the proof is computed inside your own engines, where you can verify it independently. Paridity scopes the claim honestly instead of making that trade silently.

What covers the insider case. Deliberate dataset tampering by a privileged insider is an access-control and audit problem, addressed by controls rather than by the aggregate:

Your own database access controls and change auditing on source and target - writing production data is already a privileged, audited act under your existing SOX/ICFR controls. Paridity's proof complements those controls; it does not replace them.
Paridity's own records are adversarially hardened even though the proof gate is not: evidence is append-only; the audit log is hash-chained, written through a single enrichment chokepoint, and sealed with signed checkpoints; evidence bundles are signed, hash-chained, and verifiable offline (see the audit & compliance proof). An insider could aim to fool a future proof; they cannot silently rewrite what past runs recorded.
RBAC separation (four roles, API-enforced) with actor and IP attribution on every administrative action inside Paridity.

If your threat model requires adversarial-grade tamper evidence over the databases themselves, that calls for database-side controls (WORM storage, native database auditing) - Paridity sits alongside them as the replication- correctness control, and its own evidence trail is built to the adversarial standard those controls expect.

The evidence¶

Every row below is backed by a passing, reproducible test vector or a live end-to-end run - not just a code review.

Claim component	Evidence source	Result
Canonicalization produces byte-identical concat strings across engines	The cross-engine byte-verified allowlist is the complete 66-pair C(12,2) matrix (12 engines since ClickHouse, 2026-10-08); `tests/hashing/test_vectors.json` tv011-tv022	Verified at the canonicalization (concat-string) level for all 12 pushdown engines - PostgreSQL, MySQL, Snowflake, Databricks, SQL Server, Oracle, Trino, BigQuery, Redshift, SAP HANA, DB2 LUW, ClickHouse - via live per-engine byte-identity test classes. Programmatically counted: 66 entries, 12 engines, no missing or extra pair.
Every one of the 55 engine pairs of the matrix as it stood at the 2026-08-23 run (the 11 ClickHouse pairs came later and are not part of that run) is verified directly, end-to-end - not only proven transitively via the reference implementation	Artifact `docs/live_gates/pairing_matrix_latest.{json,txt}` (run `2026-08-23T13:33:47Z`, `all_55_direct: true`)	Verified - the pairing-matrix lane ran all 55 pair configs through the real pipeline (API → Temporal → readiness → SQL → diff → evidence) in one recorded run: 55/55 native `pass_clean` with `fallback_used=False` (`hash_method=xor_sum_sha256`), each fail-case leg reaching `fail_count` (except `bigquery_to_redshift`, PASS-only - neither engine is harness-injectable). No pair false-failed on identical seeds.
`row_hash` computed via each engine's native SQL is byte-identical to the Python `HashService` reference, not just string-identical	The live byte-identity suites `tests/integration/connectors/test_cross_engine_byte_identity.py` (one `Test<Engine>ByteIdentity` class per engine; its own 33-case vector list - 29 ids drawn from `tv001`-`tv035`, not every id in that range, plus the four astral `tv048`-`tv051` - includes a NULL decimal, a zero-fraction timestamp and non-ASCII text) and `test_clickhouse_byte_identity_2827.py`	Verified live for all 12 pushdown engines - each class executes the engine's own `row_hash` SQL (the exact expression `extract_keys` emits) and asserts the hex equals the reference SHA-256 of the canonical row string. DB2 LUW, once run by hand outside the suite because of a host driver limitation, is now a gated class in the same suite, run from the worker container where the driver is installed. The full required-mode live-gate run of 2026-10-28 (`docs/live_gates/run_20261028T151921Z.json`, also `latest.json`; verdict GREEN, 1,614 passed, 0 failed, 0 errors across 12 engines) records DB2 on its own under `per_engine.db2_luw`: 88 passed, 0 failed, 0 errors, 2 skipped, across four classes - `TestDb2ByteIdentity` (32 passed, 1 skipped), `TestDb2DecryptByteIdentity` (40 passed, 1 skipped), `TestExtractRowsForHashByteIdentity` (8 passed) and `TestPreNullChainLiveDb2` (8 passed). Both skips are the same by-design case, named in `skipped_tests` with the reason: `tv017` (32 integer digits) exceeds DB2's 31-digit DECIMAL and is not representable. The per-test junit for that run is `run_20261028T151921Z.junit.xml`; the earlier hand-run per-vector record `docs/live_gates/db2_byte_identity_20261027T1333Z.txt` (DB2 11.5.9.0, 32 vectors passed at concat and hash level) agrees with it. Concat-string parity alone does not imply hash-level parity, so the suites assert at the hash level for every engine.
`hash_total` aggregate arithmetic (XOR + SUM, including the SUM-only path) is correct at defined edge cases	`tests/hashing/test_vectors.json` `ht001`-`ht010`	Verified - covers single-row, empty dataset, XOR-cancellation, row-count mismatch, SUM-modulo-boundary, the `xor_is_real=False` SUM-only comparison path (`ht009`), and the aggregate over the four astral-text row hashes (`ht010`).
Where an engine cannot render a value byte-faithfully in SQL, it refuses the native path rather than hashing different bytes	Connector refusal sites, each pinned by unit tests and recorded in the cross-engine hash verification record (Issues #68, #69, #70, fixed 2026-10-10/11): Trino `to_hex` and BigQuery `TO_HEX` over a text column under a binary plan (`NotSupportedError` after one cached schema read); SAP HANA `BINTOHEX` over an NVARCHAR column (the same refusal - HANA would otherwise accept it silently and render CESU-8); the SAP HANA row concat hashed as `STRTOBIN(..., 'UTF-8')`, never `TO_VARBINARY` (CESU-8 false-failed every astral character against every peer); the PostgreSQL row hash as `sha256(convert_to(concat, 'UTF8'))`, never `(concat)::bytea` (which parsed bytea escape input - a false pass of `\101` against `A`); PostgreSQL arrays and MySQL `json`, and `metadata_only` on any semi-structured column, always compared on the reference	Verified - every refusal lands on the DuckDB reference path with the reason logged, and the run is never a silent pass under a different byte stream. The HANA defect was found by astral (non-BMP) text, and the astral vectors `tv048`-`tv051` were added to the standing vector set so every engine is gated on that class; the PostgreSQL defect was found while disproving an unrelated array-render hypothesis - a reminder that the record is kept by testing claims, not by assuming them.
Every connector can extract raw rows for the DuckDB reference (fallback) path and produces the same `hash_total` as every other connector on a shared fixture	Live byte-identity across all 12 connectors on a shared fixture	Verified - all 12 connectors produce the identical `hash_total` (`4a136562c19cd220:4e2c14af69caec14`) on the shared 6-row fixture, live byte-identity-verified against `HashService` for every type-fidelity-bearing engine.
Cloud warehouse pairs are hash-level verified live, not only in CI fixtures	Live cloud-warehouse PASS + FAIL runs	Verified - PostgreSQL↔Snowflake, ↔Databricks, ↔BigQuery, ↔Redshift each ran a live PASS (`pass_clean`, native pushdown) and a live FAIL (`fail_count`, mismatch=1) against a hardened dataset containing CJK text, a NULL decimal, a foreign `timestamptz` source type, and uppercase target columns.
Accuracy is verified at production data volume, not only in small test fixtures	Live cross-engine runs to 100,000,000 rows	Verified - three flagship pairs (PostgreSQL→Snowflake, PostgreSQL→Databricks, Db2 LUW→BigQuery) are each verified by live reconciliations from 1M through 100,000,000 rows, and Oracle→Snowflake through 10M; every clean run was preceded by a fail-first proof in which a planted 0.01% field tamper and planted deletions were recovered exactly (timings and conditions in the companion performance proof document). Volume is also where subtle defects surface: an earlier 1,000,000-row MySQL↔Snowflake run caught Snowflake hex aggregate components that were not fully zero-padded - a genuine at-volume false-failure risk, since fixed - and in the runs cited here Oracle ran full-strength SHA-256 `DBMS_CRYPTO` pushdown with matching cross-engine `hash_method` - the weak 32-bit `ora_hash_sum` fallback, which engages when the `DBMS_CRYPTO` grant is missing (see Limitations), was never engaged in these runs. A 2026-08-31 audit of PostgreSQL ±infinity dates and timestamps found and fixed a third such class: the driver's own codec collapsed `infinity` onto the `9999-12-31` SCD2 sentinel in both directions, and `TO_CHAR` rendered an infinite timestamp as the null token - so a stored `infinity` could compare equal to a stored literal. ±infinity are now distinct values canonicalized as PostgreSQL's own text render, with a live pin.
SQL Server, Oracle, and Trino are hash-level verified by live end-to-end runs, not only by byte-identity fixtures	Live PASS + FAIL runs (SQL Server UTF-8 hashing, Oracle target-side field resolution + UTC session TZ, Trino NULL-decimal rendering)	Verified live (`pass_clean` + `fail_count`) for each. Because concat-string parity does not imply hash parity, hash-level assertion is mandatory for every engine, not just the canonicalization string.
SUM-only comparison (engines without a native XOR aggregate) is statistically sound, not just "good enough"	Collision-probability analysis	Verified - collision probability quantified at ≈1.08×10⁻¹⁹ per comparison for the SHA-256 SUM path; the weak 32-bit `ora_hash_sum` / `checksum_agg_sqlserver` fallback paths (≈2.3×10⁻¹⁰ per comparison) are flagged as weak in the proof evidence and treated as advisory, not audit-grade.

Verify it yourself¶

These steps reproduce the claim in your own environment. Steps 1-4 can be run self-serve by a technical evaluator with operator access; step 5 is independent of the running application and can be run by anyone with the repository.

Create a Mapping between two engines from the Verified list below (e.g. PostgreSQL → Snowflake), pointed at a dataset you control.
Run it once cleanly. Confirm the run reaches pass_clean and that the Run detail view shows a native-pushdown proof (not a DuckDB fallback, unless one of the two engines requires it - see Limitations).
Force a known, single-cell mismatch on the target - change one value in one row directly in the target system.
Run the Mapping again. Confirm:
The proof phase's hash_total values now differ between source and target (visible on the Run detail page).
The run's status reaches Issue detected - terminal state fail_field, because you changed a value (row counts still match, so the proof hash_total fails, then the diff phase isolates the field). Had you instead added or removed a row, it would reach fail_count. Either way - never a silent pass.
The Evidence tab's field_diff record identifies the exact row and column you changed, with the source and target values shown side by side (subject to your masking configuration).
Run the shipped test vectors independently of the running application. tests/hashing/test_vectors.json contains the canonical row-hash vectors (tv001-tv051) and hash_total vectors (ht001-ht010) that gate every engine implementation before it can be declared production-ready. These are plain input/expected-output pairs - you do not need to trust Paridity's own test runner to check them; you can independently compute SHA-256 over the documented canonical string for any vector and confirm it matches expected_row_hash.

Tip: Steps 1-4 require a live source and target you're willing to write a test value into - most evaluators do this with an SE in a sandboxed environment rather than a production system. Step 5 requires no Paridity environment at all - it can be done offline against the JSON file alone.

Limitations¶

Accuracy verification depth is not identical across every engine. This table states, honestly, what "Verified" means for each scope - over-stating this would be exactly the kind of over-claim this document exists to prevent.

Scope	Status	Notes
PostgreSQL, MySQL, Snowflake, Databricks, Trino, BigQuery, ClickHouse (native XOR aggregate available)	Verified	`xor_is_real=True` - real XOR and SUM components, native SHA-256 pushdown, hash-level byte-identity vs the `HashService` reference, including cloud pairs. ClickHouse (added 2026-10-08) is byte-verified on 26.8 and on the 25.8 LTS floor; a server below 25.8 is refused at connect time, never run on a fallback.
Runs whose schema could not be verified, or whose read was at lower fidelity than stored	Verified - never an unqualified clean pass	Schema introspection is tri-state per side (`verified` / `unavailable` / `failed`); a run with an unverified side records it in the run's drift detail, displays as "Pass · Schema unverified", and is excluded from pass counts (filterable via `display_status=schema_unverified`). Likewise a column the connector reads at lower fidelity than stored - today Trino `delta_lake` reads a Delta `TIMESTAMP` rounded to the millisecond, and Trino `hive` does the same at its default `hive.timestamp-precision` - is recorded on the run, displayed as "Pass · Reduced timestamp fidelity" and excluded from pass counts (`display_status=reduced_fidelity`). Neither label is configurable away.
Oracle, SQL Server, Redshift, SAP HANA, DB2 LUW (v11.1+)	Verified - SUM-only by design	`xor_is_real=False`: no native `BIT_XOR` aggregate on these engines, so `hash_total` comparison uses the SUM component alone; the XOR component is a fixed `0000000000000000` and is never compared. This is an architectural limitation, not an unverified gap - the SUM-only collision probability is quantified and documented (≈1.08×10⁻¹⁹ per comparison). Native pushdown is byte-verified live for all five engines.
DB2 LUW v10.5 / v11.0	Verified - via DuckDB reference fallback	These versions lack `HASH(expr, 2)`; `hash_total()` and `extract_keys()` raise `NotSupportedError` and Paridity automatically falls back to the DuckDB reference implementation (ground truth) for both proof and diff phases. Slower than native pushdown, not less accurate.
Object Store, File Dataset connectors	Verified	No SQL engine to push down to - the DuckDB reference path is always used. File Dataset: byte-identity plus live pairings. Object Store: inventory-level byte-identity vs an independent SDK listing live against S3-compatible storage, orchestrator legs, and a live end-to-end manifest-vs-bucket run (`pass_clean` + a same-length content tamper caught as `fail_field`), with per-flavor content-fingerprint constraints pinned by tests (multipart/ADLS/GCS etag semantics).
Salesforce	Verified	API-only source; always uses the DuckDB reference path. Live end-to-end verified against four target warehouses (PostgreSQL, Snowflake, Databricks, BigQuery) - `pass_clean` and `fail_field` on injected drift, with full datetime-string canonicalization. Verified via permanent pairing harness legs: Campaign and Account faithful replicas, clean pass plus injected-drift catch, DATE/DATETIME/CURRENCY/INT/PICKLIST coverage.
Oracle `ora_hash_sum` fallback / SQL Server `checksum_agg_sqlserver` fallback	Supported - advisory only, not audit-grade	Only engaged when SHA-256 pushdown is unavailable. 32-bit hash, collision probability ≈2.3×10⁻¹⁰ per comparison - non-negligible at volume for a compliance platform. Flagged as weak in the proof evidence; Paridity does not present these results as equivalent to the SHA-256 path.
Adversarial tampering of the compared datasets by a construction-aware insider	Out of scope of the proof aggregate - by design, stated explicitly	See Threat model. The `hash_total` proof targets replication faults and corruption; a privileged insider deliberately writing compensating rows into the compared datasets is covered by compensating controls (your database access controls and change auditing, plus Paridity's append-only evidence, hash-chained audit log, and signed bundles), not by the aggregate.
Cross-engine pairs where source and target `hash_method` values differ	Verified (by automatic fallback)	Paridity detects the mismatch and forces both sides to the DuckDB reference path rather than attempting a native comparison it cannot guarantee - this is the platform failing safe, not a gap. In the current engine set every proof connector emits `xor_sum_sha256`, so this path is a safety net rather than the common case.
Mappings with a declared SCD2 scope	Verified - scoped; PostgreSQL and Snowflake only	A mapping may declare a current-rows-only scope for an SCD2/bitemporal side. The proof then covers that side's current rows, not the whole table, and the run is badged "Scoped: SCD current rows" so a scoped pass is never displayed as a whole-table pass. The scope is always operator-declared and versioned inside the checksummed config snapshot - never inferred, because inferring that one side is a superset by design is a false-pass machine - and is rejected outright on any engine whose scope rendering is not live byte-verified (today `postgres` and `snowflake`; fail-closed at both validation layers). Every scoped run first checks the scope's own premise: exactly one current row per business key (always), and optionally no overlapping validity intervals. A violation reclassifies the run SCD invariant violated and excludes it from pass counts.

This document reflects the audit state as of last_verified above. New engine pairs and deeper verification depth are added continuously - the live status of every open item is available to reviewers on request through your Paridity account contact.