Lakehouse tables - Iceberg-aware reads¶

Reading a Delta Lake table without Databricks (Trino, Snowflake Delta Direct, or no SQL engine at all)? See Delta Lake outside Databricks.

An Iceberg (or Delta) table is not a new kind of system. It is a table you read through a system you already have: Trino, Databricks or Snowflake. "Iceberg-aware" means the reader also looks at the table's commit history, so Paridity compares the table as it stood at one exact commit instead of "whatever is there when the query happens to run".

In the mapping form. On the Watermark & CDC step pick the strategy commit_snapshot for a side, then its Commit source (iceberg, delta or column; column also asks for the column name). The strategy is offered only for Trino and Databricks systems and for Delta and Iceberg file datasets (a file_dataset system with flavor delta or iceberg) - the readers that have a commit reader - and each offers only the sources it can read (Trino: iceberg, delta; Databricks: delta, column; Delta file dataset: delta; Iceberg file dataset: iceberg). The same settings are available through the mapping API (see Watermarks and windows).

What to configure, per reader¶

Reader	What you register	Status
Trino	A Trino system whose catalog is an Iceberg catalog (for example a REST catalog). The platform does not store the catalog name: it comes from the `catalog` key in the secret your Connection Reference points at (default `hive`), or from the `catalog.schema.table` name of the table in the mapping. The catalog itself lives on the Trino server, and Test Connection shows the catalog in play. Refer to the table as `catalog.schema.table`.	Live-verified 2026-10-07, including a CI gate that must pass on every change.
Databricks	A Databricks system. Use a Unity Catalog managed Iceberg table by its normal three-part name, with `source = delta` (see below).	Reading the table (byte-identical to the same rows in a Delta table), the watermark and the pin are live-verified at the connector level, and the end-to-end runs passed on the platform (2026-10-07; pass_clean on native pushdown, an injected row gives fail_count, a one-character change gives fail_field).
Snowflake	A Snowflake system. A Snowflake-managed Iceberg table (`CATALOG = 'SNOWFLAKE'`) lives on an external volume in your object store, which an account administrator creates in your Snowflake account and grants to the role Paridity connects with (USAGE). Refer to the table by its normal name, and use an `ingest_ts` watermark or the Snowflake change token (see below).	Reading the table (byte-identical to the same rows in a native table) and the pin are live-verified at the connector level, and the end-to-end runs passed on the platform (2026-10-07; pass_clean on native pushdown, fail_count, fail_field). Readiness pins every run to Snowflake's clock (#2881).

On Snowflake there is no Iceberg snapshot watermark: commit_snapshot with source iceberg, delta or column is refused there with a configuration error, because Snowflake exposes no Iceberg snapshot id or sequence number to order commits by (checked again live on 2026-10-07). On the Snowflake side use either:

an ingest_ts watermark (a timestamp column set at load completion - never a business date); or
commit_snapshot with source snowflake_commit_time (#2868). It reads Snowflake's SYSTEM$LAST_CHANGE_COMMIT_TIME, a change indicator, not a timestamp. Set the watermark domain to sequence (a time_utc domain is refused). Each run compares the whole table, and runs only when the token has moved since the last successful run. The token also moves on a DML that changes no rows (a zero-row DELETE, a no-op UPDATE): that costs an extra run, never a different verdict. Its behaviour was live-verified on a Snowflake-managed Iceberg table (2026-10-12). Only the target side's token decides whether a run happens. Every run is still pinned on Snowflake. Readiness records Snowflake's own current time as the run's pin, and every query of the run reads the table AT(TIMESTAMP => <pin>). See "What pinned snapshot means" below. A Snowflake Iceberg table accepts fewer column types than a native table: text is STRING (a length such as VARCHAR(255) is refused) and timestamps are TIMESTAMP_LTZ(6) or TIMESTAMP_NTZ(6) (microseconds; TIMESTAMP_TZ is refused). Both compare exactly as the same values in a native table. On Databricks a Unity Catalog managed Iceberg table uses source = delta: Databricks keeps its commit history as table versions (DESCRIBE HISTORY) and pins a run with VERSION AS OF, exactly as for a Delta table. source = iceberg is refused on Databricks, because Databricks SQL exposes no Iceberg snapshot id to order or pin by; source = column also works.

The commit_snapshot watermark in operator terms¶

commit_snapshot answers "what is the newest committed state of this table?". It has three forms, chosen by source:

`source`	Reads	Use it when
`iceberg`	The table's snapshot history (Trino)	An Iceberg table, read through Trino.
`delta`	The Delta table history (Trino, Databricks)	A Delta table, or a Databricks managed Iceberg table.
`column`	`MAX(field)` of a column you name (`field` is required)	The table carries a version column. This form is sequence only.

delta_version is kept as an alias of this watermark; existing mappings are not rewritten.

What "pinned snapshot" means¶

At readiness, Paridity records the newest commit of each side - its pin. Every query of the run (counts, hash total, key extraction, field diff) then reads the table as of that commit, never "latest". The lag recheck is the one exception: it reads the latest data on purpose, because it asks whether a missing row has arrived since.

On Snowflake there is no commit id to pin by. When readiness lets a run go ahead, it records Snowflake's own current time instead, for every Iceberg table. That covers Snowflake-managed Iceberg tables, Delta Direct tables and tables from other external catalogs. Every query of the run then reads AT(TIMESTAMP => <that time>).

Reading at a past time uses Snowflake Time Travel, so the table needs a retention of at least 1 day: - on a Snowflake-managed table, set DATA_RETENTION_TIME_IN_DAYS to 1 or more; - on an externally catalogued table, Snowflake derives the retention from the table's metadata (history.expire.max-snapshot-age-ms).

When the pin cannot be read, the run is refused, never read at latest: at readiness as a configuration hold, or as a system error in a run that outlives the retention. Native (non-Iceberg) Snowflake tables are pinned the same way when their DATA_RETENTION_TIME_IN_DAYS is 1 or more (a retention-0 table reads latest in each phase); a view is pinned only when the mapping's Pin Snowflake views at readiness switch is on. See the Snowflake engine page, "Readiness read pin (time travel)".

Two consequences you will see:

A commit that lands after readiness is not compared in this run. It is picked up by the next run. This is deliberate: it stops a table that is still being written from producing a mismatch between the proof phase and the diff phase of the same run.
The comparison is repeatable. Re-reading the same pinned commit gives the same data.

Why a compaction does not trigger a run¶

Advancement is measured on the newest commit that changed data, not the newest commit of any kind. A commit counts as changing data unless its operation is known to leave the rows untouched: Iceberg replace (compaction), and for Delta OPTIMIZE, VACUUM, REORG and metadata-only operations such as SET TBLPROPERTIES, ADD COLUMNS or UPGRADE PROTOCOL. Anything else, including an operation the platform does not recognise, counts as new data. The pin is still the newest commit of any kind.

If a reader reports no operation names, every commit counts as data-changing: the worst outcome is an unnecessary run, never a wrong verdict.

What is recorded¶

Each readiness evaluation writes the pin (the Iceberg snapshot id or Delta version; on Snowflake, the pinned time) beside the watermark values in the append-only watermark observation (run.run_watermarks, columns source_watermark_pin_value / target_watermark_pin_value). The run page and the API do not display it today - it is evidence an auditor can read from the table.

Statuses you will see¶

No new data after a compaction. The run is skipped because no data-changing commit arrived. This is not a failure; see Runs.
A full sweep instead of a windowed run. A table-level snapshot watermark (source = iceberg or delta) marks no individual row with its commit time, so Paridity cannot cut a row window. It compares the whole table at the pinned commit. This holds in both domains (sequence and time_utc). A run still only happens when the data-changing boundary has advanced. Only the column form produces a real row window.

Domain¶

iceberg and delta can run in time_utc (translated through the commit timestamp, so a time-based source can pair with a lakehouse table) or in sequence. column is sequence only; saving it with another domain is refused. Commit timestamps are held as ISO-8601 UTC instants; a reader that returned a text rendering would be refused rather than parsed.

Known behaviour: UniForm metadata lag¶

A Databricks UniForm table publishes its Iceberg metadata asynchronously after the Delta commit, so an Iceberg reader can briefly see an older commit than the Delta side. Measured on the platform's test workspace (2026-10-07): the Iceberg commit lands about 2 seconds after the Delta commit (1.8 to 2.1 s for one-row writes, 3.1 s for a 400-file write) and is visible to a reader 4 to 5 seconds after the write returns.

Declare the mapping's Table Semantics as Databricks UniForm Iceberg mirror (cdc_table_semantics = uniform_iceberg_mirror, mapping level only, never inferred) and each run is held - not_ready, cause uniform_metadata_lag, recorded on the watermark observation - until UniForm's own delta-version table property on the Iceberg side has reached the Delta version (so a Delta commit that changes no data, such as VACUUM, releases without a new Iceberg snapshot; the newest-commit-instant comparison is only the fallback when the property cannot be read, and a pair where neither can be read holds), then released on the normal readiness retry. The setting needs an Iceberg commit_snapshot watermark on one side and a Delta one on the other (refused otherwise). With the default (none) nothing is held: a run started inside the lag window may compare an older Iceberg commit and fail on a row that is only in flight.

VACUUM and property changes (#2867). A Delta commit that changes no data creates no new Iceberg commit; UniForm only records the new Delta version in the Iceberg table properties. The hold releases on exactly that property, so such a commit no longer blocks runs: once delta-version catches up the run ends no_new_data.

Verification. The two readers of the same UniForm table were proven to produce the same hash total at the connector level, and a mirror one commit behind was proven to differ. The end-to-end run passed on the platform (2026-10-07, scripts/verify_2853_uniform_hold_live.py --prove-can-fail --vacuum-step). The hold was observed while UniForm lagged one version and released once the mirror caught up, ending pass_clean. After a VACUUM the run ended no_new_data with no hold. The same in-flight write under none failed with fail_count.

Reading a Databricks table as Iceberg through Trino¶

To read a UniForm (or managed Iceberg) table as Iceberg, give Trino a second Iceberg catalog that points at the Unity Catalog Iceberg REST endpoint:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=${ENV:DATABRICKS_UC_ICEBERG_URI}
iceberg.rest-catalog.warehouse=<Unity Catalog catalog name>
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.token=${ENV:DATABRICKS_UC_TOKEN}
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region=<the bucket's region>

The URI is https://<workspace host>/api/2.1/unity-catalog/iceberg-rest (the older /iceberg path answers 400). warehouse is the Unity Catalog catalog name.
The token never goes in the catalog file. Trino reads it from the environment (${ENV:...}) when the catalog loads. It is Trino server configuration - Paridity never sees or stores it; the platform's Trino system still stores only the catalog name.
Table data is read with short-lived storage credentials that Unity Catalog hands out per table ("vended credentials"). That needs, on the Databricks side: external data access enabled on the metastore, the table in a catalog or schema with its own storage location (a schema in Databricks default storage can never be read externally), and EXTERNAL USE SCHEMA granted to the token's principal.
Refer to tables by catalog.schema.table (for example uc_iceberg.sales.orders). SHOW TABLES on this catalog fails (Unity Catalog does not support listing views); reading, DESCRIBE and the snapshot history work.
The Paridity dev stack ships this catalog as uc_iceberg with placeholder values, so the stack starts without Databricks access; set DATABRICKS_UC_TOKEN and DATABRICKS_UC_ICEBERG_URI to use it.

What is not there yet¶

Item	State
Real partition-level reuse (re-hash only the changed Iceberg partitions)	Built (#2847). A full run re-hashes only the Iceberg partitions whose manifest set changed since the last run (live: 10 of 10 re-hashed on the first run, 8 reused on the next, and an injected row in one partition is still caught).
Snowflake `commit_snapshot`	Built (#2868): source `snowflake_commit_time`, sequence domain, the run pinned to the readiness time (#2881). Source `iceberg` / `delta` / `column` stay refused on Snowflake.
A form field for `commit_snapshot`	Built (#2884): strategy plus commit source on the Watermark & CDC step; Trino, Databricks, Delta file datasets (#2906) and Iceberg file datasets (#3190), and Snowflake with source `snowflake_commit_time` (#2868).
UniForm lag hold	Built (#2864) and proven end to end (#2853). Releases on UniForm's `delta-version` property, so a VACUUM or property change no longer blocks runs (#2867).
Reading an Iceberg table with no SQL engine (DuckDB)	Built (#3171, ch2 of epic #223): a `file_dataset` system with flavor `iceberg` reads rows through the DuckDB `iceberg` core extension and table metadata (snapshot list, schema, format version, delete files) through `pyiceberg` - either a REST catalog (`namespace.table`) or table-root no-catalog mode (`version-hint.text`). The `commit_snapshot` pin (source `iceberg`) applies on every read surface the same way it does for Trino and Databricks above. Proven at connector level 2026-10-18 (tri-identity against the Trino `iceberg` reader's native `hash_total` over the same table); end-to-end pairing legs passed on 2026-10-19 (#3172). Storage is S3 or local paths; GCS and ADLS are refused by name until a gate leg proves an Iceberg table on them.
Equality deletes on the engine-less reader	Partly built (#3328). Read on unpartitioned format v2 tables when the equality columns are `int`, `long`, `string`, `decimal`, `date`, `timestamp` or `timestamptz` and the table's metadata shows no NULLs in them (proven against Trino on every read surface). Refused by name otherwise: partitioned tables and NULL-bearing columns (DuckDB 1.5.6 gets both wrong - it removes rows it must keep), `float` / `double` and other types, and per-partition full-run reuse. Use the Trino path for those tables.
Format v3 on the engine-less reader	Partly built (#3328). Unpartitioned v3 tables read, whether created as v3 or upgraded from v2. Refused by name: nanosecond timestamps (no canonical rule for sub-microsecond digits yet), column default values, deletion vectors, partitioned v3 tables, and the `variant`, `geometry`, `geography` and `unknown` types.
Snowflake full-run partition reuse	Decided REFUSED, not built (#3169). Snowflake exposes no operator-visible partitions, and the only candidate whole-table signal (`SYSTEM$LAST_CHANGE_COMMIT_TIME`) is not a proven-complete content marker — the vendor is silent on `ALTER TABLE ... SWAP WITH`, `TRUNCATE`, and `CREATE OR REPLACE TABLE`. Every Snowflake full run does a full scan.