Paridity Architecture Technical Brief¶

Verified data. Trusted outcomes.

CDC-Aware Reconciliation Platform

Prepared for: Customer, partner and procurement evaluators (Customer Confidential) Version: v0.2 - Date: 2026-10-27

Executive summary¶

Paridity's architecture makes three commitments that most reconciliation tools cannot: the entire platform runs inside your own cloud account/VPC, so credentials, object names, and data values never leave your environment; a single vendor-neutral connector contract with cross-engine canonical hashing lets it compare heterogeneous source/target pairs (e.g. Oracle against Snowflake) natively, with a DuckDB reference engine as ground truth; and the platform is engineered for corporate-scale datasets - 100M-10B rows per Mapping - with bulk-only I/O and hash pushdown enforced as design rules, not aspirations. This brief describes how the architecture delivers each commitment and is explicit about what is roadmap rather than shipped. It is the architecture companion to the Paridity Security Whitepaper; security controls are covered there and are linked, not restated, here.

Problem¶

An enterprise architect evaluating a reconciliation platform is really evaluating three risks at once. First, data-residency risk: conventional SaaS reconciliation tools require exporting database credentials, schema/object metadata, and often the data values themselves to a vendor-hosted environment. Second, lock-in and coverage risk: tools built around one warehouse or one replication vendor cannot prove correctness across a heterogeneous estate (SAP ECC on Oracle or DB2 replicating to Snowflake, Databricks, or ADLS). Third, scale risk: an architecture that compares tables by pulling rows to a central service collapses at billions of rows. Paridity's architecture is designed around all three: customer-hosted deployment, a vendor-neutral engine model, and pushdown-first comparison that moves fingerprints, not rows.

Approach / architecture¶

Deployment model - customer-hosted, in your own cloud account¶

Today's shipping architecture. The full Paridity stack deploys inside the customer's own cloud account/VPC: the FastAPI control-plane API, the React/Vite UI, the Temporal orchestrator, all six Temporal workers (scheduler, readiness, SQL, diff, evidence, notification), and a single PostgreSQL instance holding all four schemas. Deployment is via a shipped Helm chart and Terraform reference modules for AWS, Azure, and GCP that provision the cluster, a managed Postgres instance with backups/PITR, the customer's own secret backend, and an egress-locked network - node egress deny-by-default, reaching cloud services over private paths (AWS PrivateLink endpoints, GCP Private Google Access, Azure NSG service tags) rather than the internet, with only the operator's declared source/target endpoints allowed outbound. Verified live on all three clouds. Paridity operates no part of this infrastructure - there is no Paridity-hosted control plane, no Paridity-visible copy of customer data, and no Paridity credential store in this model. This is the architecture, delivered and CI-gated - the entire platform runs inside the customer's own cloud account.

Four schemas, one separation of concerns¶

The platform's own PostgreSQL database is partitioned into four schemas whose concerns are never mixed:

Configuration schema - configuration, versioning, and audit. Holds the 7-layer config model, frozen resolved-config snapshots, and the audit log. The config_checksum (SHA-256 of the resolved config JSON) lives here - it fingerprints policy, never data.
Execution schema - execution lifecycle: runs, execution requests, watermarks, CDC state, schema-drift detection, per-phase attempt records with timestamps.
Evidence schema - append-only evidence: hash_total proof results, row_hash mismatch keys, field_diff output, run metrics, in-flight recheck records, and hash-chained evidence bundles. Evidence tables are never updated or deleted outside the audited retention purge - which now runs on a built-in schedule and skips any scope under a legal hold - and two narrowly-scoped lifecycle exceptions.
Projection schema - derived, recomputable projections (run summaries, hourly metric rollups, baselines) that dashboards read instead of raw evidence. Anything rewritable lives here by construction, so the evidence schema's append-only rule holds with no exceptions to remember - and it is actively enforced by a live row-version check plus a static code scan, not left to convention.

The separation is enforced by classification: every table and column is classified as control-plane (governance/metadata) or data-plane (sensitive - credentials, object/schema/column names, resolved snapshots, evidence values), and the two are never mixed in one table.

Orchestration - six workers on dedicated task queues¶

All execution flows through Temporal (self-hosted, MIT-licensed) durable workflows. Work is divided across dedicated task queues - scheduler, readiness, SQL, diff, evidence, and notification - so that pushdown SQL, in-memory diffing, and evidence writes scale independently. Workers execute single activities and contain no orchestration logic; connectors execute SQL and contain no reconciliation logic; comparison logic lives in workers and services. Data moves between workers and the DuckDB diff engine as Apache Arrow - a columnar interchange format, not row-at-a-time serialization.

Controls / details¶

Vendor-neutral engine model¶

Every source and target system is accessed through one contract: the BaseConnector abstract base class. Connectors are deliberately dumb adapters - they execute SQL and return data; all comparison logic lives above them. This is what keeps Paridity vendor-neutral: adding an engine means implementing one interface, not re-architecting the comparison pipeline.

Native hash pushdown with a reference fallback. For the proof phase, Paridity asks each engine to compute the hash_total - the aggregate dataset fingerprint - inside the engine itself, using that engine's native SHA-256 facility (e.g. sha256() + BIT_XOR on PostgreSQL, SHA2 on Snowflake and Databricks, DBMS_CRYPTO on Oracle, HASHBYTES on SQL Server, HASH(expr, 2) on DB2 LUW 11.1+, HASH_SHA256 on SAP HANA, SHA256 + groupBitXor on ClickHouse). Only fingerprints cross the network, never rows. Where an engine cannot compute a compatible hash, the connector raises NotSupportedError and the orchestrator falls back to the DuckDB reference implementation - the ground-truth engine every native implementation is byte-verified against - rather than silently computing something different.

Cross-engine canonical hashing is the enabling mechanism. Heterogeneous comparison (Oracle source, Snowflake target) is possible because every column value is normalized to an identical canonical string form - null token, fixed-point numerics, UTC ISO-8601 timestamps, \x1F-delimited row serialization - before hashing, so identical data hashes identically regardless of which engine produced it. Source and target must use the same hash_method; if they differ, both sides fall back to the DuckDB reference. The per-engine verification record - which pairs are byte-verified live, which are SUM-only, and every open accuracy issue - is maintained in the platform's cross-engine hash verification record and distilled for evaluators in How Paridity guarantees accuracy; this brief points to that record rather than restating it.

Engine coverage today: native xor_sum_sha256 hash pushdown on twelve engines - Oracle, DB2 LUW (11.1+), SQL Server (2019+), Snowflake, Databricks, BigQuery, Redshift, SAP HANA, MySQL/Aurora MySQL, PostgreSQL, Trino (covering ADLS/Hive external tables), and ClickHouse (25.8 LTS and later; an older server is refused at connect, never run on a fallback). Object-store and file datasets, and Salesforce, are compared via the DuckDB reference path (no SQL engine to push down to); Kafka CDC participates in readiness evaluation, not dataset proof. Replication-tool integration is likewise adapter-based: Debezium, Airbyte, Fivetran, Oracle GoldenGate and Qlik Talend Cloud are the Tier-1 (production) adapters; the design-partner-pending stubs are exactly SAP SLT, SAP BODS, Qlik Replicate (with its legacy attunity alias) and Informatica CDC - Roadmap per the status vocabulary below, not presented as shipped. Any other tool in the catalogue has no adapter and is Evaluating.

CDC-aware windowing - comparing at the same point in time¶

Most reconciliation platforms compare tables. Paridity compares datasets within a deterministically aligned time window, using CDC tool state to avoid false positives from in-flight replication. A readiness evaluation - CDC probe, watermark fetch, latency evaluation, stability check, advancement check - runs before every comparison and follows an explicit state machine. Watermark strategies are selected in a defined preference order (control_marker first, snapshot last), and Paridity never infers freshness from business dates such as posting date - it uses replication metadata, CDC state, or explicit snapshot mode. Two storage-engine behaviours that can move the picture after readiness are handled as declared mapping settings, never guessed: a Snowflake native table with Time Travel retention is read AT(TIMESTAMP) of the readiness instant (views opt in with read_pin_views; a no-retention table is read unpinned and logged as such), and a mapping may declare cdc_table_semantics - uniform_iceberg_mirror holds readiness until the Iceberg side has published the compared Delta commit; replacing_merge_tree_final and clickpipes_peerdb read a ClickHouse ReplacingMergeTree side through FINAL (the latter also filtering the PeerDB soft-delete marker) so unmerged row versions cannot false-fail a run.

Systems without a native change timestamp - SAP and similar¶

Many SAP application tables carry no column that advances on every insert, update, and delete. Paridity treats that as a supported configuration rather than an exception. A watermark strategy is chosen per mapping from a defined preference order: control_marker (a completion record your pipeline writes when a load finishes - preferred), delta_version / kafka_offset / oracle_scn / lsn / db2_commit_ts (a CDC offset or commit sequence), ingest_ts (a load-completion timestamp), partition_max, timestamp (a fallback - it infers rather than observes), custom (an operator-reviewed SQL expression), and finally snapshot or partitioned_snapshot.

Two rules are absolute. Paridity never infers freshness from a business date: a posting date or document date records when something happened in the business, not when the row arrived, so a backdated or corrected document would fall outside every future window and never be re-compared. And auto-configuration never selects a column as a change watermark unless its name identifies it as a tracking timestamp; any other candidate is surfaced for operator review and is never applied silently.

Where no trustworthy signal can define a complete window, the mapping runs as an explicit snapshot. snapshot compares the whole current dataset on every run; partitioned_snapshot scopes each run to partitions declared on the mapping (by key range, value list, time grain, or a composite of two - for SAP typically client, company code, or fiscal year), and a schedule with window_mode = full provides the periodic full sweep. In both modes the readiness phase has nothing to fetch - no watermark, latency, or stability check is evaluated - and the run proceeds straight to the same count, hash-total, mismatch-key, and field-diff proof as every other run, with the comparison mode recorded in the run's evidence.

Declared scope for history-bearing tables¶

An SCD2 or bitemporal side legitimately retains closed history rows the other side no longer shows, so comparing the two whole fails forever on data that is correct. Paridity resolves this with an operator-declared scope, never an inferred one: a typed current-row predicate (a flag column, a far-future end-date sentinel, or a NULL end date - columns and values, never operator SQL) on one named side, versioned inside the checksummed resolved-config snapshot. Inferring that one side is a superset by design would be a false-pass machine, so the platform refuses to infer it, and refuses the declaration outright on any engine whose scope rendering is not live byte-verified. Every scoped run first checks the scope's own premise - exactly one current row per business key, and optionally no overlapping validity intervals - before the proof. A broken premise never silently passes: the run keeps its data verdict, is reclassified SCD invariant violated, and is excluded from pass counts. Watermark queries are deliberately never scoped: the current-row boundary is mobile, so scoping it would hide a pure logical delete behind a "no new data" verdict.

Scale & performance - engineering commitments, not marketing numbers¶

Paridity is engineered to remain viable at corporate scale - 100M-10B rows per Mapping - and treats throughput as the gate immediately after accuracy. The following are binding engineering rules enforced in code review, and are presented here as design commitments distinct from benchmarked results. Measured results now exist at production scale and live in the Paridity performance proof document: four flagship cross-engine pairs measured end-to-end on a published reference configuration - three of them (PostgreSQL→Snowflake, PostgreSQL→Databricks, Db2 LUW→BigQuery) through 100M rows - each preceded by a fail-first corruption-recovery proof. This brief cites those measurements rather than restating them.

No row-by-row I/O on the hot path. Reads use server-side cursors with bulk prefetch (asyncpg cursors with prefetch ≥ 10,000, Oracle array fetch, Snowflake Arrow-batch streaming, Databricks Arrow fetch); staging writes use bulk APIs (PostgreSQL COPY FROM STDIN via copy_records_to_table). Row-at-a-time INSERT loops are forbidden on extract/stage paths, with a per-engine bulk pattern defined for every SQL engine (ClickHouse streams keys as Arrow batches).
Pushdown is the default, fallback is the exception. Hash-total and key-extraction work runs inside the source/target engine wherever the engine supports it; every fallback to the DuckDB reference path is logged with its reason.
Throughput floors (per worker pod, 4 vCPU baseline) are defined as merge gates: ≥ 200,000 rows/sec for proof (hash-total) pushdown on Tier-1 engines; ≥ 25,000 rows/sec for PostgreSQL/MySQL key extraction; ≥ 50,000 rows/sec for Snowflake/Databricks key extraction. A change that drops below a floor is treated as a regression and must be fixed before merge. Measured reference points now sit well above these floors - PostgreSQL key extraction has been measured at ~507,000 rows/sec at 1M rows on the reference configuration (see the Paridity performance proof document); the floors remain the merge gate, not the marketing number.
The platform's own database is protected the same way. Hot-path queries must be index-backed; dashboards read pre-aggregated summary tables, never raw evidence rows in-request; evidence tables are always queried with a run_id predicate and explicit column projection.

Reliability¶

Durable workflows. Each run executes as one Temporal workflow instance per execution request - never two for the same request. Workflow state survives worker restarts; scheduling uses Temporal's durable timers, not an external cron daemon.
Explicit state machines. Execution requests, runs, and readiness evaluations each follow a defined state machine; every status transition goes through the defined paths and terminal transitions are written to the audit log - no ad-hoc status updates anywhere in the codebase. A deterministic configuration refusal discovered after readiness (a connector rejecting the mapping's declared types, a partition plan that cannot be executed) terminates in its own config_error state - operator work, raised as a warning - and is never conflated with system_error, the fault terminal that pages on-call.
Idempotency at two layers. All state-mutating API endpoints require an Idempotency-Key header with a 24-hour server-side response cache, and duplicate execution requests for the same mapping+window are rejected by a database UNIQUE constraint on the idempotency key - not silently ignored.
Evidence writes cannot be quietly lost. Evidence-write activities run under a dedicated retry policy (10 attempts, no non-retryable error types); long-running activities are heartbeat-monitored.
Audit anchors are immutable. The run's compare-start audit timestamp is set once when a run starts and never changes - not during in-flight rechecks, not on reruns. In-flight rechecks happen inside the same workflow and each iteration is recorded as recheck evidence.
Retention is executed, not merely configured. A durable Temporal singleton on the scheduler worker runs the retention purge on a cadence (daily at 04:00 UTC by default), writing one audit attestation per fire. It is on by default but dry-run-first for the first seven days after install, so a fresh deployment never deletes on day one, and legal holds are evaluated on every leg - automatic deletion always skips a scope under a preservation order.
Backpressure, not overload. Per-system concurrency is enforced through a database-backed counter with max_concurrency resolved as the minimum across config layers - each system's profile, source and target, is a hard ceiling no schedule or run can exceed.

Security controls (summary - see the Security Whitepaper)¶

Authentication (OIDC/JWT), RBAC, tenant isolation, secrets handling (connection_ref is always a URI pointer resolved via SecretsClient against the customer's own secret backend - Paridity never stores a credential), evidence masking, and the append-only hash-chained evidence trail are covered in depth, with the same Verified/Supported/Roadmap grading, in the Paridity Security Whitepaper. One caveat is repeated here because it is architectural: PostgreSQL row-level security is live as a defense-in-depth backstop - the application-layer tenant predicate on every query is the primary isolation mechanism today, because the application's database role currently has BYPASSRLS and FORCE ROW LEVEL SECURITY is intentionally deferred pending a call-site audit (see the Security Whitepaper, "Tenant isolation").

Proof / evidence¶

Paridity's correctness claim is deterministic end to end: aligned time windows, an aggregate hash_total as the fast pass/fail proof gate, per-row row_hash fingerprints to isolate exactly which rows differ, and field_diff output as the reconciliation evidence. Every native hash implementation is byte-verified against the DuckDB reference engine, and the per-engine verification status - including open issues, which are never closed without live test vectors - is tracked in the platform's cross-engine hash verification record and presented for evaluators, with a verify-it-yourself procedure, in How Paridity guarantees accuracy. Every run records its config_checksum to the append-only audit evidence before the proof phase begins, so an auditor can independently confirm which policy was in force.

Note: AI in Paridity assists triage and explanation only - it never determines whether data matches. Correctness is always established by the deterministic hash_total / row_hash / field_diff pipeline above, never by a model's judgment.

Limitations & roadmap honesty¶

Capability	Status	Notes
Customer-hosted deployment - full stack in customer VPC	Verified	Helm + Terraform, packaging delivered and CI-gated.
Live `helm install` / `terraform apply` validation on a real cluster/cloud account	Verified	From-empty EKS, AKS, and GKE installs with fail-first signature verification, plus Azure and GCP `terraform apply`, live-attested against the published release artifacts. CI gates `helm lint`/`template` and `terraform fmt`/`validate` for all three clouds, plus a compose smoke of the production images, on every change to those surfaces; `terraform plan`/`apply` are live-attested in the committed install transcripts, not run in CI.
Vendor-neutral connector contract (`BaseConnector`) across all listed engines	Verified	Single ABC; per-engine verification depth tracked in the cross-engine hash verification record.
Native cross-engine hash pushdown	Verified per pair	Every native-pushdown engine pair in the 66-pair cross-engine matrix is byte-verified, and the full matrix has additionally been run end-to-end to native clean passes - see the cross-engine hash verification record and the accuracy proof doc for per-pair depth and the SUM-only engine caveats.
Replication adapters - Debezium, Airbyte, Fivetran, Oracle GoldenGate, Qlik Talend Cloud	Verified	Tier-1 production adapters. Fivetran live-verified against a real account: adapter read-path plus reconciliation end-to-end. GoldenGate live-verified on a real GG Free 26ai replicating pair: three-verdict profiled e2e (pass_clean / no_new_data / fail_count), MA REST and classic_heartbeat modes. Qlik Talend Cloud live-verified on a live trial tenant (RDS PostgreSQL → Snowflake): the same three-verdict profiled e2e, CDC probe consulted on every run.
Replication adapters - SLT, BODS, Qlik Replicate, Informatica CDC	Roadmap	Design-partner-pending stubs; contract defined, not production-validated.
Throughput floors (≥200K rows/sec proof pushdown; ≥25-50K rows/sec key extraction)	Supported	Binding design commitments enforced as merge gates; presented as engineering rules, not published benchmark results.
Published benchmark sizing matrix (1M-10B rows, per engine pair)	Partially measured	Four flagship pairs are measured on the reference box (see the performance proof doc): PostgreSQL → Snowflake, PostgreSQL → Databricks, and Db2 LUW → BigQuery at 1M / 10M / 100M rows, and Oracle → Snowflake at 1M / 10M (100M is out of scope on the disclosed free-edition Oracle cap). All 1B and 10B cells remain deliberately unmeasured for every pair.
Declared SCD2 scope for history-bearing tables	Verified - PostgreSQL and Snowflake	Scope rendering is live byte-verified on those two engines; every other engine rejects the declaration rather than applying it on faith.
Tenant isolation - PostgreSQL RLS as sole enforcement	Roadmap	RLS live as defense-in-depth; `FORCE ROW LEVEL SECURITY` deferred pending call-site audit (see Security Whitepaper).
Security certifications (SOC 2 / ISO 27001), independent penetration test	Roadmap	Covered honestly in the Security Whitepaper - no certification or third-party pentest exists today.
Cost & Usage - the cloud cost a comparison incurs	Scan volume measured on eleven engines; stated as Not measured, never estimated, elsewhere	See the statement below.

What Paridity can and cannot tell you about cost¶

Paridity reports the additional cloud cost you incur because it ran a comparison - never a total cost of operation, and never a guess. What it can say, and the limits it states on the page:

Scan volume is measured, not modelled, on eleven engines - Snowflake, Databricks, BigQuery, file datasets, Qlik Cloud, Trino/Hive and ClickHouse, whose drivers or servers return bytes scanned per statement (ClickHouse on the proof statements; a streamed key extraction there is recorded as unmeasured, never as zero), plus Oracle, SQL Server and Redshift (#2456) and Db2 LUW (#2457), where Paridity reads the figure from the engine's own catalogue immediately after each proof statement. Those four bill by instance/RPU/cluster hours or per core, so their bytes are a calibration denominator rather than a vendor unit. On every other engine the page says Not measured and states how many mappings its figures cover; it never renders an unmeasured estate as $0.00 or ranks one it cannot see. PostgreSQL is a deliberate Not measured: every per-statement source it offers is either aggregated across connections, instance-wide, or costs a second execution of the proof query - and no PostgreSQL estate bills per byte (Aurora bills I/O requests, which are not bytes).
Every dollar is your rate × recorded bytes, frozen at run time. A charge appears only for a system you gave a rate to (scheduled change → system → tenant default → otherwise $0). Each run's cost is written as append-only evidence at the rates in effect when it started; changing a rate never rewrites history, and the page footnotes the runs that are not frozen.
Egress is billed per side - source bytes at the source system's rate, target bytes at the target's. The suggested egress rate is 0 - configure one only if your provider bills you for result traffic, that is, your topology crosses regions or clouds. On the reference Snowflake and Databricks estates neither vendor recorded a transfer charge for in-region query results.
A $/TB rate is only true for the cadence it was measured at. Warehouse-billed engines charge for uptime; at the reference rigs' short-run cadence the measured effective rate was tens of times the list ballpark. Paridity ships a calibration procedure that derives your rate from your own bill, and does not publish a portable rate. Every rate is stored with its calibration provenance - the date, cadence, rows per run, bill source and attributable share of the window it was measured in - shown beside the number, and is flagged stale when the observed cadence has drifted more than 2× from the calibrated one.
"Last 30 days" is the cost incurred in the trailing 30 days - a measured total, not a forecast. Paridity does not project spend.

Appendix¶

Paridity Security Whitepaper - the security companion: access control, secrets, data protection, supply chain, DR.
How Paridity guarantees accuracy - the accuracy claim, verification method, and per-engine status.
Paridity glossary - canonical terminology (hash_total, row_hash, field_diff, config_checksum, Mapping, Replication Profile, watermark).
Supporting internal references - the deployment split-readiness model and data-classification manifest, the three state-machine references, the BaseConnector connector contract, the cross-engine canonicalization and hashing algorithm, the per-engine hash verification record and open accuracy issues, the hardware baseline and reproducible benchmark harness, and the full dependency/license inventory - are available to reviewers on request through your Paridity account contact.
Follow-up questions: route through your Paridity account contact or the Trust Center (in progress).