How Paridity stays efficient at scale - and what we have actually measured¶
The claim¶
Paridity establishes whether two datasets match without transferring either of them, and the volume of data it moves afterwards is proportional to the size of the difference, not the size of the dataset. A clean 100-million-row reconciliation moves a row count and a short fingerprint string from each side - and nothing else.
Every performance figure presented here as a result is a measurement taken on named hardware, at a named row count, by a named harness, at a named commit. No result in this document is extrapolated, modelled, or projected, and none is a contractual commitment. One extrapolation does appear, in Limitations, shown specifically to demonstrate why we do not publish them. Where we have not measured something, this document says so rather than estimating it.
That second paragraph is part of the claim, not a disclaimer around it. A reconciliation platform that overstates its own throughput has the same defect as one that overstates a match: it produces a number that gets acted on and is not true.
How it works (deterministic)¶
Efficiency in Paridity is not the result of tuning. It comes from doing the cheap test first and only paying for the expensive one when the cheap test fails.
- The proof gate runs inside your engine. Paridity asks the source and
target databases to each compute a
hash_total- a single aggregate fingerprint over every row in the window - using their own native SQL. What crosses the network is a row count and a fingerprint pair per side, on the order of tens of bytes. For a dataset that matches, this is the entire cost of the reconciliation: no rows are read out of either system, and there is no egress charge to pay. - The comparison window is aligned before anything is computed. Paridity uses watermark state - a CDC offset, a control marker, or an explicit snapshot - to pick a window in which both sides are comparable. This is an accuracy mechanism first, but it is also an efficiency one: comparing mid-replication produces a false difference, and a false difference costs a full diff cycle to disprove.
- Only the rows that could differ are ever extracted. When the proof gate fails, Paridity does not pull both tables. The optional stratified proof splits the aggregate into 256 independent buckets, so a failing run can tell which buckets disagree and extract only those. A changed row dirties at most two buckets, so on a sparse difference the extraction is a small fraction of the table.
- Only the rows that actually differ are compared column by column. Row
fingerprints (
row_hash) identify the differing rows; thefield_diffphase then runs per-cell comparison on those rows alone. - Pushdown is the default, and every fallback is logged with its reason. The hash and key-extraction work runs inside the customer's engine wherever the engine can express it. Where it cannot - an engine without a native SHA-256 aggregate, or a transform whose semantics differ across engines - Paridity falls back to its own reference implementation and records why. The fallback is slower; it is never less accurate.
The consequence worth internalising: the expensive phases only run when something is wrong. A healthy estate is cheap to prove continuously.
Note: AI never determines correctness in Paridity. Correctness is always established deterministically - by aligned windows,
hash_totalaggregate fingerprints,row_hashper-row fingerprints, andfield_diffcolumn comparison. AI assists triage and explanation of already-computed results only; it never decides whether data matches, and it plays no part in any figure on this page.
The reference hardware¶
Every figure published as a result below was taken on one machine, referred to throughout as the reference box - with three exceptions, each labelled where it appears and each explained at the end of this section. Numbers were captured with the machine otherwise idle, and with the three CI runners confirmed not busy first. Where a figure was taken with the worker process deliberately capped at 4 CPUs on that box (the ClickHouse key-extraction readings), the cap is stated beside it.
| CPU | AMD Ryzen AI 9 HX 470 - 12 cores / 24 threads |
| Memory | 64 GB |
| Storage | Samsung SSD 990 PRO 2 TB NVMe |
This is our hardware, not a model of yours. A 4 vCPU worker pod - the baseline our internal engineering floors are written against - will produce lower absolute figures, and a DuckDB path that uses 24 threads here will not find 24 threads there. Where a figure depends on thread count, this document says so.
No throughput or duration figure measured on a developer laptop or on the Docker development stack is published as a result in this document. Such measurements exist internally, and on the laptop they run at roughly half the box across the board, which is why they are not a basis for anything told to a customer.
There are exactly three exceptions, each marked at the point it appears:
- The narrowing egress cut, in the evidence table below. It is a ratio of rows extracted to rows extracted, which is hardware-independent by construction - how many rows a query returns does not depend on how fast the machine running it is.
- The readiness short-circuit timing, in the floors table further down. It is a genuine development-stack duration and it is labelled as one. It appears because the margin against its target is four orders of magnitude, which no change of hardware closes; it is presented as evidence that a floor is met, never as a performance figure to quote.
- The Databricks partition-reuse comparison, in its own subsection below. Its durations were recorded on the development stack against a cloud serverless warehouse and are labelled as such. It appears because the point it establishes is a comparison between two code paths on the same run shape, and the per-row gap between those paths (a Python reference hash at tens of microseconds per row against a native pushdown at tens of nanoseconds) is three orders of magnitude, which no change of worker hardware closes. The durations are not performance figures to quote.
The evidence¶
Each row states the row count it was taken at and the commit at which it was measured. Every one of these harnesses drives the real production code path - the actual Temporal activity or the actual connector method - not a reimplementation of it. The internal records behind each figure are available to reviewers on request through your Paridity account contact.
Proof phase - establishing a match without moving data¶
| What was measured | Result | Scale and conditions |
|---|---|---|
PostgreSQL hash_total native pushdown |
434,204 rows/sec (230.31 s) | 100,000,000 rows, reference box idle, 2 alternating repeats, commit 60057634 |
| The same path, independently re-run at a different commit | 433,657 rows/sec (230.60 s) | 100,000,000 rows, same rig, same day, commit 610f86a4-era |
PostgreSQL hash_total at smaller scale |
922,680 rows/sec at 1M; 945,781 rows/sec at 5M | 1,000,000 and 5,000,000 rows, reference box, commit 7fde8a9b |
| Engine-less proof path, vectorized | 29,909 → 345,646 rows/sec (11.6×) at 1M; 29,470 → 400,185 rows/sec (13.6×) at 5M | 1,000,000 and 5,000,000 rows, 12 columns, 8% nulls, reference box, commit 58940a72 |
| Stratified proof - the cost of localizing a failure | +32.9% over the plain proof (305.98 s vs 230.31 s) | 100,000,000 rows, reference box, commit 60057634 |
The 100M-row figure is the one to hold on to: it is the largest-scale proof measurement we hold, and the two independent runs agree to within 0.13%. Note that it is materially lower than the 1M and 5M figures above it - see the note on extrapolation in Limitations, which is the single most important caveat on this page.
The vectorized row matters for a different reason: it is the path used when
neither side is a SQL engine - files, object stores, Qlik - where there is
nothing to push down to. That path runs at 1.7× our internal proof floor. It is
enabled by default for the file_dataset and qlik_cloud connectors and
nowhere else, because those are the two surfaces where it has been byte-identity
verified at volume.
Key extraction - the first phase that reads rows¶
Key extraction runs only when the proof gate fails. Each side streams its row fingerprints into the platform's staging store; every figure below is the shipped code path, and where the span includes that staging write it says so.
| What was measured | Result | Scale and conditions |
|---|---|---|
PostgreSQL key extraction (stream_keys) |
507,375 rows/sec at 1M; 531,727 rows/sec at 5M | 1,000,000 and 5,000,000 rows, reference box, commit 7fde8a9b |
| Keyset-paginated extraction, completeness asserted | PostgreSQL 637,265 rows/sec; MySQL 575,420 rows/sec | 200,000 rows each, reference box, commit 7fde8a9b; every key returned exactly once |
| Databricks target-side extraction, full activity including the staging write | p50 55,990 rows/sec (57,624 / 55,990 / 55,769; spread 3.3%) | 10,000,000 rows x 20 columns, reference box worker against a 2X-Small serverless warehouse, 3 fail-path runs (ef9cba2e / eb4cdf57 / 003661ab), box commit ec5d516a |
| Snowflake target-side extraction, including the staging write | 123.9K rows/sec (PostgreSQL source side 95.2K rows/sec on the same run) | 100,000,000 rows x 20 columns, reference box worker against an X-Small warehouse, one fail-path run (d0fc9aae), commit a8d6eba7 |
ClickHouse extraction through the worker's staging write (_stage_side_copy) |
145,642-152,167 rows/sec at 50,025 rows; 157,217-163,567 rows/sec at 2,000,000 rows | 50,025 rows (7 columns) and 2,000,000 rows (3 columns), reference box with the bench process capped at 4 CPUs, five uncontended runs across ClickHouse 25.8.33.6 and 26.8.20.9, each a median of 5 after a warm-up, commits 70910123 (runs 1-2 per server) and 04f14672 (run 3) |
The Databricks and Snowflake rows are the target-side half of fail-path runs from the flagship campaigns further down, isolated per side from the activity history rather than from a micro-benchmark, and the span is the whole extraction activity - stream, canonicalise, write to staging. The Databricks figure clears its internal floor by 1.12x on three runs. The Snowflake figure is a single run and is recorded here as such, not certified. The ClickHouse rows carry a can-fail control: the same bench with its process throttled to 0.1 CPU read 17,299 rows/sec, reported BELOW FLOOR and exited non-zero.
Difference phase - cost proportional to the difference¶
| What was measured | Result | Scale and conditions |
|---|---|---|
| Field diff, as shipped today | 503,712 cell-pairs/sec (25,186 keys/sec) | 1,000,000 keys × 20 columns, 1% of cells differing, reference box idle, commit c6b54e3a |
| Field diff, cumulative improvement across three changes | 222.77 s → 37.34 s = 5.97× | 1,000,000 keys × 20 columns, reference box, 6a61110f → 5b9884a6 |
| Connections opened by a 1M-key diff | 4,000 → 2 | 1,000,000 keys, reference box |
| Egress avoided by narrowing on a sparse difference | 12.7× fewer rows extracted (20 of 256 buckets) | 100,000 rows / 10 planted differences, commit 84d5bdbd - see note below |
On the field-diff unit. A cell-pair is one source cell compared against one target cell, and the denominator counts cells processed, including the ones that turn out to be equal. Any rows/sec restatement must carry the column width, because the same run has a different rows/sec at every width. The measurement above restates as 25,186 rows/sec at 20 columns; our internal floor for this phase, discussed below, restates as ≈7,500 rows/sec at the same width. Those two figures are often confused - one is what we measured, the other is the minimum we will accept. We do not quote a width-free rows/sec number for this phase, and neither should anyone quoting us.
On the narrowing row. This is the one measurement here not taken on the reference box - it was taken on the Docker development stack. It is included because it is a ratio of rows extracted to rows extracted, and how many rows a query returns does not depend on how fast the machine running it is. The cut is also scale-independent by construction: it is a function of how many of the 256 buckets contain a difference, which depends on the density of the difference and not on the row count. The measurement bears that out - 100,000 rows and 300,000 rows with 10 planted differences gave 12.7× and 12.8×. At high difference density the benefit disappears and the run degrades gracefully to full extraction, measured at 1.0× with no penalty.
Concurrency and platform overhead - the cost is your engine, not the platform¶
| What was measured | Result | Scale and conditions |
|---|---|---|
| Paridity's own CPU cost while a reconciliation runs | 0.8% of CPU (engine ~1202%, the whole platform stack ~9% on its isolated partition) | Reference box, PG→PG, 2M × 20-col slices, engine pinned to 16 vCPU, concurrency N = 1 → 30, cpuset hard-partition isolating platform CPU from engine CPU, 2026-08-23 |
| Aggregate throughput vs engine vCPU | ~9.5-10K rows/sec per engine vCPU, ~linear (8 vCPU → 80.8K rows/s; 16 vCPU → 152.5K rows/s) | Same rig, cpuset-partitioned |
| Enterprise-pattern concurrency, error-free | 20 fully-overlapping reconciliations - 0 errors, 0 throttles, 0 lost evidence across 78 runs, every run pass_clean on native pushdown; submission→start queue ≤ 0.24 s, ~2 s non-proof overhead per run even at N = 20 |
Reference box, single worker-container stack, PG→PG 2M × 20-col, 2026-08-17 |
What this establishes, and what it does not. Paridity's own orchestration cost stays flat and sub-second as concurrency rises - the platform is not the bottleneck, and its overhead is a fraction of one percent of CPU. The per-run slowdown visible at high concurrency in these tests is the test engine saturating, not Paridity - both sides of all N mappings deliberately shared one stock-configured 128 MB Postgres container, a fraction of any enterprise database. Aggregate reconciliation throughput is bounded by the source/target engine capacity you size, and scales roughly linearly with the vCPU you give it - a property you provision, not one this platform caps. A single large reconciliation saturates about 8 engine vCPU and then flattens (one proof query parallelizes to ~4 workers), so the way to use more capacity is to run reconciliations concurrently. The figures in this section are PostgreSQL-to-PostgreSQL. Four flagship cross-engine pairs have been measured end-to-end on the same reference configuration, under shipped defaults, each preceded by the fail-first protocol (a planted 0.01% field tamper and planted deletions recovered exactly before any clean number was accepted):
| Flagship pair | Proof p50, 1M rows | Proof p50, 10M rows | Proof p50, 100M rows |
|---|---|---|---|
| PostgreSQL → Snowflake | 13.1 s | 79.9 s | 12.5 min |
| PostgreSQL → Databricks | 12.7 s | 79.5 s | 12.6 min |
| Oracle → Snowflake | 12.5 s | 108.8 s | not measured - the Oracle source used for these runs is the free edition, capped at 2 CPU threads / 12 GB (disclosed deliberately-unflattering hardware); 100M is out of scope on that cap |
| Db2 LUW → BigQuery | 52.5 s | 8.5 min | 85.9 min |
Rows are 20 columns wide throughout - a rows-per-second figure is meaningless without the width, so it is stated. Every cell is the p50 of three cache-busted clean runs with run ids recorded in the internal benchmark register; where a source engine is the bottleneck (Oracle's 2-thread cap, Db2's in-database SHA-256 rate) that is disclosed rather than averaged away.
Flagship cross-engine pair - Db2 LUW → BigQuery, measured to 100M rows¶
This is the most heterogeneous of the four measured pairs - a Db2 LUW source reconciled against a BigQuery target, from one million to one hundred million rows, on the reference box (one worker container, no CPU cap recorded, against BigQuery on-demand) - and the pair documented in most depth here. Every clean number below is preceded by a fail-first proof: a 0.01% field tamper and a batch of planted deletions were each recovered exactly before any clean timing was accepted.
| Rows (20 columns) | Proof phase (p50) | End-to-end, clean | BigQuery data scanned per proof |
|---|---|---|---|
| 1,000,000 | 52.5 s | ~55 s | ~297 MB |
| 10,000,000 | 8.5 min | ~8.7 min | ~3.1 GB |
| 100,000,000 | 85.9 min | ~87 min | ~29 GiB |
Each tier is three cache-busted clean runs with a tight spread (0.2%-1.0%), plus corroborating fail-path runs. The long pole is the source engine, not the platform: Db2's native SHA-256 hash, computed in-database at its measured optimal parallel degree, runs at roughly 19K rows/sec on this 20-column shape and dominates every tier; the BigQuery side and Paridity's own orchestration are a small fraction of the wall clock. Proof duration and BigQuery bytes scanned both scale close to linearly with row count. When a proof fails and the platform extracts and diffs the mismatched rows, the full end-to-end on a 0.01% mismatch is key-extraction bound - roughly 2 minutes at 1M, 24 minutes at 10M, and 2.6 hours at 100M - and every planted corruption at every tier was recovered exactly.
100M rows is the largest measurement held; 1B and 10B are deliberately unmeasured and are established on a customer-engagement basis, not projected from this range.
Full-run partition reuse on Databricks - measured at 10M rows, and slower than the plain proof below a break-even¶
Paridity has an opt-in optimisation for full-table runs on partitioned
lakehouse tables (fullrun_reuse_enabled, default off): keep the per-partition
proof of the previous run and re-hash only the partitions the table's change
feed says moved. It was measured before any default was considered, and the
measurement went against it. This is the third labelled exception: the
durations are development-stack runs against Databricks' serverless warehouse,
single runs each, and they are published as a comparison, not as throughput.
Two identical 10,000,000-row Delta tables (4 columns, 12 partitions, change
data feed on), one row appended to one partition per side between the first
and second run, every run pass_clean, proof durations read from the
platform's own phase record:
| Run shape | Rows hashed | Before (#3192, recorded at commit 062d9117, run-tag 20261023131118) |
After (#3238, measured at commit 3efc0d27, run-tag 20261024155805) |
|---|---|---|---|
| Plain native pushdown, reuse off (control) | 20,000,002 | 10,250 ms | 9,225 ms |
| Reuse, first run - no baseline, every partition re-hashed | 20,000,000 | 779,431 ms (76x the control) | 62,675 ms (6.8x) |
| Reuse, 1 of 12 partitions changed per side | 1,666,668 re-hashed, 18,333,334 reused | 79,670 ms (7.8x) | 13,809 ms (1.5x) |
The "before" column found the cause: a changed partition was re-hashed through the Python reference implementation at about 41 microseconds per row, while the plain path it replaced is the engine's native hash at about 45 nanoseconds per row above a fixed cost of roughly 9 seconds. The "after" column is the fix - a changed partition is now re-hashed through the engine's own scoped native hash, so the combined result stays byte-identical to a full scan - and it is still slower than the plain proof at this shape, because listing the changed partitions has a fixed cost of several seconds that a 10M-row native scan does not pay. The harness's fit puts the break-even at roughly 14M skipped rows per side on this warehouse class; that break-even is a two-point projection and is labelled indicative, not measured. The consequence is already in the product: the setting stays off by default, and its help text says it is slower than a plain full run below that break-even.
Why these numbers can be trusted¶
This section is the actual differentiator, and it is worth more than any individual figure above.
- The benchmarks that carry the load have been proven able to fail. A green
benchmark that has never been seen to fail is not evidence. Two of the
harnesses behind the figures above ship a
--prove-can-failmode that plants a specific defect and requires the run to go red: the field-diff harness (five legs) and the vectorized-hash harness (two planted defects plus an untampered control). Two more carry a can-fail control of a different kind: the ClickHouse key-extraction bench was throttled to 0.1 CPU and seen to report BELOW FLOOR and exit non-zero, and the frontend first-paint gate was seen to fail on the build it was introduced against (425.5 KB) before it passed on the fixed one. The others - the PostgreSQL floor, evidence-write, PostgreSQL and MySQL key-extraction, stratified and narrowing benchmarks - do not have a documented can-fail proof; several were established before we adopted that requirement. That is a real gap in our evidence and we are not going to describe it as anything else. - Every performance run is simultaneously an accuracy run. The field-diff
benchmark asserts that the set of
(key, column)cells written to evidence is exactly the planted set - count, membership, and no duplicates - and the 1M-key certification run above cleared that on 200,126 planted cells. Every stratified proof run is gated on the recombined hash being byte-identical to the plain hash, on both repeats. The vectorized proof harness required byte-identical results on 6 of 6 lanes at both 1M and 5M rows, and separately demonstrated that all four of its fallback paths return the reference answer rather than a different one. A fast wrong answer fails these benchmarks, which is the only arrangement compatible with our accuracy posture. - Scaling behaviour is measured wherever we could measure it. The PostgreSQL proof, vectorized proof, PostgreSQL and ClickHouse key-extraction and field-diff results were each taken at two or more scales, which is what distinguishes "fast" from "fast until it isn't" - a single-point benchmark cannot. Four results are single-point and should be read as such: the stratified overhead (100M only), the field-diff certification run (1M only), the Databricks key-extraction floor (10M only) and the Snowflake key-extraction reading (100M only, one run).
- Corrections are recorded rather than quietly fixed. Where a benchmark figure has been superseded - a measurement re-run after contamination was found, or a predicted scaling regression that the benchmark later refuted - the correction stands in the record rather than being silently overwritten.
Our internal throughput floors - design gates, not commitments¶
Paridity engineering maintains throughput floors as merge gates: a change that drops a path below its floor is treated as a regression and must be fixed before it lands. They are written against a 4 vCPU worker pod.
These are internal engineering rules. They are not service levels, they are not contractual, and they are not a prediction of what you will see on your hardware against your data. They are published here for one reason: to be honest about which of them we have actually demonstrated and which we have not.
| Path | Floor (4 vCPU baseline) | Demonstrated? |
|---|---|---|
Proof hash_total pushdown |
≥ 200,000 rows/sec | PostgreSQL only. Cleared at 100M rows on the reference box (2.2×). Proof throughput has been measured for the other eleven pushdown engines, but only at 50,000-60,000 rows and only on development or cloud rigs - never on the reference box. At that size each figure is dominated by something other than per-row throughput (per-row engine cost on Oracle and DB2 LUW, network transport on the cloud engines, fixed query latency on ClickHouse), so none is a valid floor reading in either direction and none is published here. |
| Key extraction - PostgreSQL / MySQL | ≥ 25,000 rows/sec | Yes, both. PostgreSQL 507,375 rows/sec at 1M and 531,727 at 5M on the box (commit 7fde8a9b); keyset pagination at 200K rows measured PostgreSQL 637,265 and MySQL 575,420 rows/sec, each asserting every key was returned exactly once. |
| Key extraction - Snowflake / Databricks | ≥ 50,000 rows/sec | Databricks: yes, p50 55,990 rows/sec at 10M rows × 20 columns over the full extraction activity including the staging write, three runs, a 1.12× margin (box commit ec5d516a). Snowflake: one run, 123.9K rows/sec at 100M rows × 20 columns on the same scope (run d0fc9aae, commit a8d6eba7) - recorded, not certified, because a single run is not a floor demonstration. |
| Key extraction - ClickHouse | ≥ 50,000 rows/sec | Yes. 145,642-163,567 rows/sec across five uncontended runs at 50,025 and 2,000,000 rows on the box with the bench process capped at 4 CPUs, on both CI-pinned server versions (25.8, the oldest the platform accepts, and 26.8), about 3× the floor (commits 70910123 / 04f14672). The verdict was seen to go red under a 0.1-CPU throttle. |
| Field diff | ≥ 150,000 mismatched cell-pairs/sec | Yes. 503,712 cell-pairs/sec at 1M keys × 20 columns on the box, a 3.4× margin. The floor is set from this measurement. |
| Evidence writes | ≥ 50,000 rows/sec | Yes. 101,764-126,787 rows/sec across four write paths at 200,000 rows on the box, a 2.0-2.5× margin (commit 7fde8a9b). |
| Readiness, when no replication profile is configured | < 5 seconds | Cleared by a wide margin, but on the development stack - p50 3 ms, p95 146 ms over 540 recorded attempts. Not re-measured on the reference box, so the figures are not published as a result above. The margin is four orders of magnitude, so the conclusion is not in doubt even if the numbers are rig-bound. |
| CDC probe | < 5 seconds | No figure. When this document was first written the probe recorded no duration at all (2,121 attempts with no timing). The platform now records a timed cdc_probe phase row on every run where the probe actually executes, so the telemetry exists - but no measurement against the budget has been taken on the reference box, so we still cannot state a figure. |
| State machine transitions | < 50 ms | No. Never measured. |
| Frontend first-paint JavaScript | ≤ 350 KB gzipped | Yes - 313.3 KB (320,823 bytes), and this is the only performance budget checked automatically on every code change. A correction is owed here. The previous edition of this document reported 347 KB against a gate set at 400 KB; that figure counted only the entry chunk, while the browser also preloads every chunk the landing page imports statically. Counted properly, first paint had been 425.5 KB - over budget - since before the figure was published. The gate now counts the entry chunk plus every preloaded chunk, is set at the 350 KB budget itself, and was seen to fail on the 425.5 KB build (commit 03ff55d2) before the dashboard's chart library was moved off the critical path (commit 31b78329). Margin today: 36.7 KB. |
| Frontend time-to-interactive, chart re-render | ≤ 2.5 s / < 100 ms | No. Never measured. |
Three rows of that table still say we have not measured something, and the proof row says it for eleven of twelve engines. Publishing a table where every row says "met" would have been easy and would have been false.
What the platform checks at runtime, and what it does not. The floors above
are merge gates. Since the throughput_floor analyzer landed, the
key-extraction floors are also evaluated on every real run that extracts
keys: the platform computes the run's key-extraction rate - the same number the
run narrative prints - and compares it with the floor for the engine pair
(PostgreSQL and MySQL 25,000; Snowflake, Databricks and ClickHouse 50,000
rows/sec; the slower side's floor governs). A run materially below floor -
under half of it, by platform default - opens one anomaly on the mapping, and
no second one while the first stands unresolved. A run whose rate cannot be
computed (a phase shorter than ten seconds, or a side whose rows were not
counted) is reported as unmeasured, never as a pass, and never as zero. An
engine pair with no documented floor is reported as having none. The proof
and field-diff floors are not evaluated at runtime; they remain benchmark and
pre-merge gates only, and nothing in the product checks them against a real
run.
Verify it yourself¶
Efficiency is the easiest of Paridity's three proof claims for you to test, because unlike accuracy it needs no adversarial setup - you simply run the product against your own data and read the timings it records about itself.
- Run a clean reconciliation and watch what does not happen. Point a
Mapping at a dataset of at least 50,000 rows that you expect to match. When
it reaches
pass_clean, open the run and read the Proof Phase card: it shows the two hash totals and whether they matched. A clean run has no Compare Phase card at all - because the diff phase never ran. That absence is the claim in section 1 of How it works, and it is the whole efficiency argument in one screen. - Compute the throughput yourself, from the platform's own record. The Proof Phase card reports Proof DB time (both sides, run in parallel). Divide the row count shown on the Data Window card by that duration. That figure is yours - your hardware, your engines, your network - and it is the only throughput number that should inform your sizing. Note the card sums both engines, so it can legitimately exceed the run's wall-clock time.
- Make the difference sparse, then make it dense. Introduce ten
differences into a large target table and run it; then introduce thousands
and run it again. Compare the Data Volume card between the two runs, and
the volume and egress tracking at
/administration/cost-usage. This is the proportional-cost claim, and it is the one most worth confirming yourself, because it is what sets your steady-state running cost. - Confirm the failure path is not silently cheap. An efficiency claim only means something if the platform is doing the work. Check that the run which found ten differences reported exactly ten, with the right rows and columns named in the evidence. Any reconciliation platform can be made faster by checking less, so verify these two properties together or neither is worth much.
- Read the platform's own floor verdict. On a run that extracted keys,
the run narrative's Key Extraction step prints the throughput it computed
across both sides, or says why it could not ("not measured" when a side's
rows or the phase timing were not recorded). A run materially below the
floor for your engine pair opens a
throughput_flooranomaly on the mapping. The absence of that anomaly on a run whose throughput the narrative prints is the platform's statement that the floor was met on your hardware; the absence of a printed throughput is not.
Tip: Steps 1-3 need nothing from us - they read timings the platform already records about itself. For sizing a production deployment against your row volumes and engine pair, ask your Paridity contact for an SE-led benchmark against a dataset shaped like yours; the engine pair and the shape of your data matter more than any figure on this page.
Limitations¶
Stated with the same discipline as the evidence. Over-claiming here would undo the point of the document.
| Scope | Status | Notes |
|---|---|---|
| PostgreSQL proof, key extraction, evidence writes, field diff | Verified | Measured on the reference box at 200K-100M rows against the real production code paths. Verification depth is not uniform: the field-diff result carries both an exact-set accuracy gate and a can-fail proof, the stratified proof result carries a byte-identity gate on every run, and the key-extraction result asserts completeness (every key exactly once). The plain PostgreSQL floor and evidence-write results carry neither an accuracy gate nor a can-fail proof - they are throughput measurements only. |
Engine-less proof path (file_dataset, qlik_cloud) |
Verified | Vectorized path byte-identity verified at 1M and 5M rows and enabled by default for these two connectors only. qlik_cloud carries no volume measurement of its own - its engine read is page-decode bound and the throughput gain there is small; it is on the verified surface because it is verified, not because it is fast. |
| Cross-engine sizing matrix at production scale | Verified | Measured - four pairs. Four flagship pairs are measured on the reference box (see the flagship table above): PostgreSQL → Snowflake, PostgreSQL → Databricks, and Db2 LUW → BigQuery at 1M / 10M / 100M rows; Oracle → Snowflake at 1M / 10M (100M is out of scope on the disclosed free-edition Oracle cap of 2 CPU threads / 12 GB - a paid-tier Oracle source would be a separate measurement). 1B and 10B rows are deliberately unmeasured for every pair (customer-engagement basis). |
| Proof floor on engines other than PostgreSQL | Roadmap | Eleven of the twelve SQL pushdown engines have no valid measurement against the 200,000 rows/sec floor. Proof figures do exist for all eleven - MySQL, Oracle, DB2 LUW, SQL Server, Trino and ClickHouse on development rigs, Snowflake, Databricks, BigQuery, Redshift and SAP HANA on live cloud rigs - but every one was taken at 50,000-60,000 rows, and our own record warns against reading a proof figure at scales of this order as a floor result. Each is dominated by something other than per-row throughput: a per-row DBMS_CRYPTO call on Oracle, hex-to-integer arithmetic on DB2 LUW, network transport on the cloud engines, fixed query latency on ClickHouse. This cuts both ways - Redshift measured ~958K rows/sec at 60,000 rows (Redshift Serverless, 2026-07-12, result cache disabled; no commit recorded for that reading) and ClickHouse ~985K rows/sec at 50,025 rows (commit 6c24087d, development rig), five times the floor, and neither is any more publishable than a sub-floor reading, because neither is measuring what the floor measures. |
| Snowflake / Databricks / ClickHouse key extraction | Supported | Measured unevenly. Databricks is demonstrated at 10M rows on three runs; ClickHouse at 50K and 2M rows on five runs in a 4-CPU-capped process; Snowflake on one run at 100M rows. None of the three has been measured at a second scale on the reference box except ClickHouse, and ClickHouse has not been measured above 2M rows. The floor is also evaluated at runtime on every real run for these engines (see the floors section), which is a check on each deployment, not a substitute for the benchmark. |
| Full-run partition reuse (Databricks Delta, Trino Iceberg) | Supported | Measured - slower than the plain proof below a break-even. Measured at 10M rows per side on Databricks, on the development stack, before and after the re-hash path was moved to the engine's native hash. Reuse remains off by default and is 1.5× slower than the plain proof at that shape; the break-even (about 14M skipped rows per side) is a two-point projection, not a measurement. The Trino Iceberg path takes the same code route and passes its correctness drill, but carries no timing at any scale - unmeasured. |
| Runtime evaluation of the proof and field-diff floors | Not built | Only the key-extraction floors are evaluated on real runs. A deployment whose proof or field-diff phase runs below its internal floor produces no signal from the platform; those two floors are benchmark gates only. |
| Any duration figure at 1B or 10B rows | Roadmap | We have never run a reconciliation at those volumes. The 100M-row measurement above is the largest we hold. |
| Extrapolating our figures to a larger row count | Not supported | Our own record shows why. PostgreSQL proof throughput is flat from 1M to 5M rows (922,680 → 945,781 rows/sec), and extrapolating that flatness to 100M predicts roughly 106 seconds. The measured 100M result is 230.31 seconds - about half the rate. We have not established the cause (a plausible one is that 100M rows no longer fit in cache the way 5M does, but that is a hypothesis we have not tested), and the cause does not change the guidance: flatness across one order of magnitude does not license a projection across two. Use the 100M measurement; do not project from the 5M one, and do not project past 100M at all. |
| Field-diff I/O concurrency gains | Supported | Measured on PostgreSQL only. The reduction from 4,000 connections to 2 applies to every engine and is the larger win. The concurrent-fetch overlap was measured on PostgreSQL. When it was measured, all but two of the SQL connectors called their blocking driver directly inside the coroutine, so a prefetch could not start until the compare yielded; every SQL connector's keyed fetch has since been moved onto a dedicated driver thread per connection (commit 0a2bb532), which removes that structural reason - but the overlap has not been re-measured on any other engine. Do not apply the fetch-concurrency figures to a cloud warehouse. |
| Cross-region and cloud-warehouse latency figures | Not published | Internal artifacts contain a latency model built by injecting round-trip delay. It is a model and it is labelled as one; no modelled figure appears in this document. |
| API endpoint response times | Not published | Measured across sixteen endpoints and comfortably inside every internal target, but on the Docker development stack rather than the reference box, so they are excluded under this document's own rule. |
| Contractual performance commitments | None | Nothing in this document is a service level. Our support terms state our actual contractual commitments, which cover support response times and do not include throughput. |
This document reflects the measurement record as of last_verified above.
The authoritative internal record - including every figure deliberately
excluded here - is available to reviewers on request through your Paridity
account contact.