Skip to content

Delta Lake outside Databricks

A Delta Lake table is not a new kind of system. Paridity already reads Delta tables through Databricks. This page covers the three ways to read a Delta table that another engine wrote, without Databricks in the read path:

  1. Trino, through its delta catalog.
  2. Snowflake Delta Direct, a Snowflake Iceberg table whose metadata Snowflake generates from a Delta log.
  3. A file dataset in delta mode, which reads the table straight from object storage with no SQL engine at all.

All three compare a Delta table to the other side of a mapping exactly as any other pair does: a fast aggregate proof first, then a row and field diff only when the proof fails. The general idea of reading a table "as of one commit" is explained in Lakehouse tables - Iceberg-aware reads; this page assumes you have read the section on the commit_snapshot watermark there.

Pick the reader by what you need. If your Delta tables hold timestamps with sub-millisecond digits and you need them compared exactly, do not use Trino (see Known limitations). Databricks, Snowflake Delta Direct and the file dataset reader keep stored microseconds.

The three readers at a glance

Trino delta catalog Snowflake Delta Direct File dataset, delta mode
Where the proof runs In Trino (native SQL pushdown) In Snowflake (native SQL pushdown) In Paridity, on the reference path (no SQL engine)
Watermark strategy commit_snapshot, source delta ingest_ts (source delta is refused on Snowflake) commit_snapshot, source delta
Version pin FOR VERSION AS OF <Delta version> on every query of the run AT(TIMESTAMP => <readiness instant>) on every query of the run (Snowflake's own clock, recorded at readiness) The Delta version is passed to the reader on every read of the run
Where to set the watermark Mapping form (Watermark & CDC step) or API Mapping form (ingest_ts) Mapping form (commit_snapshot, source delta) or API
Timestamp precision Millisecond (rounded) - runs are labelled Reduced timestamp fidelity Microsecond Microsecond
Deletion vectors Honoured Honoured (after a refresh) Honoured
Column mapping Not covered by our tests name mode honoured name mode honoured
Storage What your Trino catalog can reach What the Snowflake external volume reaches S3 with access keys or through the AWS credential chain (IAM role / instance profile, verified on the LocalStack emulator with a profile file), GCS with an HMAC key (verified on the fake-gcs-server emulator), ADLS with an account key or SAS (verified on the Azurite emulator), or local paths (see limits)
Evidence Live-verified 2026-10-07 Live-verified 2026-10-07 Live-verified 2026-10-07

"Live-verified" means the reader was checked against a reference computation on real tables, and end-to-end runs passed on the platform (pass_clean on a matching pair, fail_count when a row was injected, fail_field when one character was changed). The evidence is in the verification record (cross_engine_hash_audit.md, Issues 59, 60 and 61).

Trino delta catalog

What you register. A Trino system whose catalog is a Delta Lake catalog (Hive Metastore and S3). The platform does not store the catalog name: it comes from the catalog key in the secret your Connection Reference points at (default hive), or from the catalog.schema.table name of the table in the mapping. The catalog itself lives on the Trino server, and Test Connection shows the catalog in play. Refer to the table as catalog.schema.table.

Watermark. Choose commit_snapshot and set Commit source to delta (Watermark & CDC step of the mapping form, or the mapping API). Paridity reads the table's commit history, records the newest Delta version at readiness as the run's pin, and reads every query of the run with FOR VERSION AS OF <that version>. A commit that lands after readiness is not in this run; the next run picks it up. The pin applies to the Delta side only.

Compactions do not trigger runs. Advancement is measured on the newest commit that changed data. OPTIMIZE, VACUUM and metadata-only commits do not count, so a compaction ends as No new data. See the Iceberg guide for the full list.

What holds. Counts, the native hash proof, key extraction and the keyed field fetch were checked against the reference, byte for byte, for strings (including non-ASCII), NULLs, decimals, booleans, dates and timestamps with millisecond-or-coarser digits. Deletion vectors are honoured on every read surface: a table with rows deleted through a deletion vector reads the surviving rows only, and a pinned read of the version before the delete reads the deleted rows again. If a deletion vector file is missing, the read fails loudly; it never returns the wrong row set.

What does not. Timestamps with sub-millisecond digits (see below).

Snowflake Delta Direct

What it is. A Snowflake Iceberg table created through a catalog integration with CATALOG_SOURCE = OBJECT_STORE and TABLE_FORMAT = DELTA. Snowflake never writes the data. Another engine (Databricks, Spark, delta-rs) writes the Delta table under the Snowflake external volume, and Snowflake generates Iceberg metadata for it. An account administrator creates the external volume and the catalog integration, and grants them to the role Paridity connects with. Register it as a normal Snowflake system and refer to the table by its normal name.

Watermark. commit_snapshot with source delta is refused on Snowflake (Snowflake exposes no Delta history). Use an ingest_ts watermark: a timestamp column set at load completion (never a business date). The Snowflake change token (commit_snapshot source snowflake_commit_time, #2868) is live-verified on Snowflake-managed Iceberg tables only; on a Delta Direct table it moves when a refresh applies a commit, including a refresh of an OPTIMIZE (observed in #2881, not gated). Because the Delta Direct table shows only what Snowflake has refreshed, a window never covers rows Snowflake has not yet seen.

Content changes only on refresh. A Delta commit is invisible to Snowflake until the table is refreshed, either by ALTER ICEBERG TABLE ... REFRESH or by auto-refresh. Plan for this: a run started between a Delta commit and the next refresh compares the older content. Measured on the platform's test account with a 30 second refresh interval, an auto-applied commit became visible after about 23, 49 and 81 seconds (three samples); a manual refresh is synchronous. A refresh that spans several commits shows only the newest.

The pin. When readiness lets a run go ahead, it records Snowflake's own current time as the run's pin. Every query of the run then reads the table AT(TIMESTAMP => <pin>): the counts, the hash, the key extraction and the field diff. A refresh that lands after readiness, including an auto-refresh, therefore cannot change the run. The next run picks it up.

The pin travels Snowflake's refresh timeline, not the Delta commit timeline: a row committed before the pin but refreshed after it is not visible at the pin. The pin instant is on the run's readiness record (the watermark pin value).

The lag recheck is the one read that does not use the pin. It reads the latest refresh on purpose, because it asks whether a missing row has arrived since.

Retention. Reading at the pin uses Snowflake Time Travel, so it needs a time-travel retention of at least one day. For a Delta Direct table, Snowflake sets that retention itself, from the table's metadata, capped at 1 day (Standard Edition) or 5 days (Enterprise Edition and above). It was 1 day on our test account. If Snowflake cannot read the pin, the run is refused and never read at latest: - at readiness, as a configuration hold that names the fix; - or during a run that outlives the retention, as a system error.

What holds. All 29 verification cases rendered byte-identically to the reference, with Delta timestamp read as TIMESTAMP_LTZ(6) and timestamp_ntz as TIMESTAMP_NTZ(6) (microseconds kept), binary through the BLOB comparison modes, deletion vectors honoured after a refresh, and column mapping in name mode honoured (a renamed column appears under its new name after a refresh).

What does not.

  • Row Tracking. A table with row tracking enabled is readable, but row tracking metadata is not surfaced.
  • Change reads. A table with Change Data Feed enabled is readable, but there is no change read: Snowflake refuses the CHANGES clause and change tracking on these tables.
  • At most 1,000 Delta commits per refresh. Snowflake's own documentation states it processes a maximum of 1,000 Delta commit files each time a table is refreshed (counted after the latest checkpoint). We took this from the vendor documentation and did not reproduce it. A table that receives many commits between refreshes, or a first refresh of a long, un-checkpointed history, may need several refreshes to catch up.
  • Other Snowflake restrictions from the vendor documentation: no INTERVAL, no decimals above precision 38, no unsigned integers, no FLOAT16.

File dataset, delta mode

What it is. A file dataset system with system_flavor = delta. Paridity reads the Delta table directly from object storage; there is no SQL engine in the path. It is useful when the table is not reachable through Databricks, Trino or Snowflake. Because there is no engine to compute the proof, it runs on the reference path inside Paridity (a run shows fallback_used true; that is by design, not a degradation). The reference path is slower than native SQL pushdown.

What you register. A file dataset system with system_flavor = delta. The mapping's object is the table root (for example s3://bucket/prefix/table; on a local filesystem, schema.table resolves to <root_path>/<schema>/<table>.delta). The root path remains a containment boundary: nothing outside it is read.

Watermark. commit_snapshot with source delta. Paridity reads the commit history with delta-rs, records the newest Delta version at readiness and passes that version to the reader on every read of the run (counts, rows for the proof, key extraction and the keyed fetch). Compactions do not trigger runs, as above. The other strategies (ingest_ts, timestamp, partition_max, delta_version) are refused, because file times and partition directories describe the Parquet files, not commits. In the mapping form, commit_snapshot (source delta) is offered for a file dataset in delta mode; file datasets of any other flavor do not offer it.

What holds. All verification cases, as UTC-adjusted timestamps and as timestamp_ntz, rendered byte-identically to the reference, and stored microseconds are kept (the reader does not round). Deletion vectors are honoured, including a pinned read of the version before the delete. Column mapping in name mode is read by logical name. A Delta table pointed at with any other file flavor (parquet, csv, ...) is refused, because reading its Parquet files directly would count files the Delta log has already removed.

What does not.

  • Storage in this release. S3 with explicit access keys in the connection secret (a custom endpoint, region and scheme are honoured); S3 through the AWS credential chain - IAM role, instance profile, profile file - when the connection secret says "aws_credential_source": "chain" and carries a region but no keys (the worker's own AWS identity is used; a secret that declares chain AND carries keys is refused, and a secret with neither keys nor the declaration is refused); ADLS Gen2 / Azure Blob with account_name plus an account_key or a sas_token (the public Azure endpoint; a custom Azure endpoint is refused); GCS with an HMAC key - the connection secret carries gcs_hmac_access_id and gcs_hmac_secret (create the key for a service account under Cloud Storage > Settings > Interoperability); and local paths. The credential-chain mode was verified on an emulator with a profile file; the instance-profile path uses the same chain but has not been run against real AWS. GCS was verified on an emulator, not against a real Google Cloud account. ADLS without an explicit key (managed identity) and GCS without an HMAC key are refused when the system connects, with an error naming the limitation.
  • Reader features. A table is refused unless its reader features are within deletionVectors, timestampNtz and columnMapping, its column mapping is none or name (not id), and its minimum reader version is 3 or lower. Other features are added only after a verification run proves them.
  • A pinned version whose files were removed. If the version a run is pinned to was VACUUMed, or no longer exists in the log, the read is refused with an error. It never silently reads the latest version instead.
  • No row filter. A filter_sql on this reader is refused.
  • Extensions are baked into the image. The Delta reader depends on DuckDB extensions installed when the worker image is built. A worker image without them reports a configuration error naming the missing extension; rebuild the image, a restart does not add them.

Known limitations

Trino reads a Delta TIMESTAMP at millisecond precision

Trino's delta catalog reads a Delta TIMESTAMP (the UTC-adjusted type Databricks writes) as timestamp(3) with time zone and rounds it to the millisecond: a stored .123456 is read as .123000, and .999500 rounds up across the second. The table on disk is not affected; other readers see the stored microseconds. TIMESTAMP_NTZ columns are not affected and keep their microseconds. The Trino documentation for its current release still lists this mapping with no precision setting, so an upgrade does not fix it.

What Paridity does about it, so you are never misled:

  • A run that reads such a column (a compared field, a key, or the side's watermark column) is recorded as read at reduced fidelity. A passing run shows as Pass - Reduced timestamp fidelity, is excluded from the pass counts on the dashboard and in metrics, and can never present as a clean pass. No setting changes that. A failing run keeps its Issue detected label, and the run detail names the column either way. The runs API filter is display_status=reduced_fidelity. See Runs.
  • The risk it guards against is a false pass: two Delta rows that differ only below the millisecond read as equal. A second shape is a false fail: a microsecond-exact source (for example Postgres or Databricks) compared with the rounded Trino read fails on that column even though the stored values match.
  • The label also covers the reference path, not only pushdown.
  • For an exact comparison, read the table through Databricks, Snowflake Delta Direct or the file dataset delta reader, or compare TIMESTAMP_NTZ columns.

Snowflake Delta Direct

Refresh-driven visibility, the readiness pin and its time-travel retention, no Row Tracking metadata, no change reads and the 1,000-commits-per-refresh vendor limit are described in the Snowflake section above. commit_snapshot stays refused on Snowflake.

File dataset delta mode

S3 (access keys or the AWS credential chain), GCS (HMAC key), ADLS (account key or SAS) and local paths only; a reader-feature allowlist; a VACUUMed pin is refused, never read as latest. See the file dataset section above.

UniForm lag, when a Delta table is mirrored as Iceberg

A Databricks UniForm table publishes its Iceberg metadata asynchronously after the Delta commit (measured: the Iceberg commit lands about 2 seconds later and is visible to a reader about 4 to 5 seconds after the write returns). If one side of a mapping reads the Delta table and the other reads its Iceberg mirror, declare the mapping's Table Semantics as Databricks UniForm Iceberg mirror. Each run is then held (not_ready, cause uniform_metadata_lag) until the mirror has caught up to the Delta version, and a Delta commit that changes no data, such as a VACUUM, releases the hold without a new Iceberg snapshot. The setting needs an Iceberg commit_snapshot watermark on one side and a Delta one on the other. Full detail is in Lakehouse tables - Iceberg-aware reads.

Where to look next