Runs¶
Navigate: sidebar Monitor → Runs (/runs). Visible to every
authenticated user. A run's detail page is also reachable directly at
/runs?run=<execution_request_id>.
A run is one execution of one mapping over one window. This guide explains what a run's conclusion means and what you can do about it. Deviations the analyzers raise about a run's trend (row count, duration, throughput) are listed separately under Monitor -> Anomalies; see Anomalies.
The two status vocabularies¶
This is the single most important thing to understand, and the most common source of confusion.
- What you see on screen is a two-part display status: a headline and a qualifier, e.g. Pass · Clean or Issue detected · Wrong values.
- What the API and the database store is an internal enum, e.g.
pass_clean,fail_field.
They map one-to-one. The display form exists because "fail_hash" is not a
useful sentence for a business reader. Note that the Evidence page's run
picker currently shows the internal value, so the same run can read
fail_field there and Issue detected · Wrong values on this page.
Complete status reference¶
Passing
| On screen | Internal | Meaning |
|---|---|---|
| Pass · Clean | pass_clean |
Source and target agree across the compared window. |
| Pass · Resolved on recheck | pass_resolved_lag |
Mismatches were seen mid-run, then every one resolved on recheck before completion - replication was simply still in flight, or (with no replication tool) one side had been read as of the readiness instant while the other read latest and both sides agree at latest. Zero outstanding at the end. |
Issues found
| On screen | Internal | Meaning |
|---|---|---|
| Issue detected · Data disagreement | fail_hash |
The aggregate fingerprints differ. The datasets disagree somewhere. |
| Issue detected · Missing rows | fail_count |
Row counts differ — rows are absent on one side. |
| Issue detected · Wrong values | fail_field |
The same rows exist on both sides but specific field values differ. The most actionable failure. |
Nothing to compare
| On screen | Internal | Meaning |
|---|---|---|
| No new data · No outstanding issues | no_new_data |
The watermark has not advanced. Nothing new arrived. Not a failure. |
| No new data · Issues resolved | no_new_data |
No new data, and previously-open issues cleared on re-check. |
| No new data · Some resolved, some still open | no_new_data |
Partial recovery of known issues. |
| No new data · Open issues | no_new_data_open_issues |
No new data and known issues persist. The platform keeps watching. |
Held, halted, stopped
| On screen | Internal | Meaning |
|---|---|---|
| Readiness hold · reason | readiness_error |
Readiness held the run and it was not compared. This is the product working, not a crash. The second part names why: Replication latency (replication was still behind after every check), Replication stalled (the target did not move at all between checks - the replication job may be stopped), Replication not running (a snapshot mapping, and the replication tool reported a connector paused, stopped or failed - see Replication profiles), Load incomplete (the load pipeline had not written its COMPLETE marker, or the target was still changing), Source unavailable (the platform could not read how far the data had loaded), Replication status unknown (strict CDC mode and the replication tool could not be checked), Configuration issue (the mapping's readiness settings are invalid). Older runs may show Source not ready. |
| Config refusal · family | config_error |
The platform refused to compare as configured after readiness had passed (or been bypassed), and nothing was compared. This is the product working, not a crash: a retry cannot change the result, and something in the configuration or the data shape needs an operator. The second part names the family: Object or connection config (a table, column or connection setting is invalid or no longer exists - a deleted table, a revoked grant), Watermark config, Filter SQL, Duplicate keys (more than one in-scope row shares a key on one side), Partition strategy (overlapping partitions, rows outside every partition, or a partition that wrote no proof), Snapshot pin unavailable, Schema unavailable, Engine requirement, Data encoding; Operator action required when the refusal has no named family yet. Refusals found during readiness show as Readiness hold instead. |
| Schema drift · Not compared | schema_drift_error |
The live schema no longer matches the accepted snapshot; a drift control fired and no comparison ran. |
| Schema drift · Available data passed / failed | any + reduced coverage | A reconciled column was excluded from this run (continue_excluding). A reduced-coverage run can never read as a clean pass — this is deliberate. |
| Pass · Schema unverified | pass_clean + unverified schema |
The data proof passed, but schema introspection failed on at least one side (permission loss, catalog error) — the drift control did not run, so the schema this run compared under was never verified. Amber, counted as an issue, excluded from pass counts. Never a clean pass, and no setting can make it one. |
| Pass · Schema not verifiable | pass_clean + unverified schema |
Same guarantee, calmer signal: the only unverified side(s) are on engines that cannot introspect at all (e.g. raw object storage) — a declared, standing property of the system, not a per-run alarm. Still excluded from pass counts. |
| Schema unverified · Not compared | schema_drift_error (halt_reason schema_unverified) |
The mapping's Unverifiable Schema policy is halt: the platform refused to compare against a schema it could not verify. |
| SCD invariant violated · Data passed / Data failed | any pass/fail + violated invariant | The mapping declares an SCD scope and this run found the scope's premise broken on the scoped side — a duplicate current row per business key, or (when opted in) overlapping validity intervals. The data verdict is preserved in the qualifier, but the run is counted as an issue and excluded from pass counts — a scope whose premise is broken cannot certify a clean pass, and no setting changes that. |
| Pass · Reduced timestamp fidelity | pass_clean / pass_resolved_lag + reduced read fidelity |
The data proof passed, but a column this run compared was read below the precision stored in the table: Trino reads a Delta Lake TIMESTAMP, or a Hive-catalog TIMESTAMP at the catalog's default precision, at millisecond precision (rounded), so two values that differ only below the millisecond compare equal. Amber, excluded from pass counts. Never a clean pass, and no setting can make it one. A failing run keeps its Issue detected label; the run detail names the column either way. |
| Cancelled · By operator | cancelled |
Someone stopped it. |
| Skipped · Schedule skipped | skipped |
The scheduler skipped this cycle. |
| System problem · System error | system_error |
A platform or connector fault — not a statement about your data. |
In flight — Running · Checking readiness, · Verifying, · Proof
passed, · Diff queued, · Comparing fields, · Rechecking in-flight rows, ·
Retrying, · Throttled, · Queued, · Starting.
Schema review pending is a separate badge, not a status. It means a new column was detected but the run still passed. Additive drift never halts — review the pending decision on the mapping when convenient.
What to do about each outcome¶
| Outcome | Your move |
|---|---|
| Issue detected (any) | Open View Evidence to see the mismatched keys and the exact differing values. Issues carry forward and are automatically re-checked on the next run, so you can watch them resolve. Use Export for Auditor to package the proof. |
| Schema drift · Not compared | The Schema Drift card lists what changed. Refresh Schema re-introspects the live tables and accepts them as the new baseline — do this once you've confirmed the change was intended. |
| Pass · Schema unverified | The run detail records which side(s) could not be verified and why (schema_verification in the drift detail). Fix the introspection fault (usually catalog permissions) — the amber clears on the next run that verifies. Filter all such runs with display_status=schema_unverified on the runs API. |
| SCD invariant violated | The run detail's invariant card shows which check failed and the count (duplicate current keys / overlapping intervals). Duplicate current keys are counted by exact key value, the same rule the comparison matches keys by: Key and key are two keys even on a case-insensitive column. This is an upstream pipeline defect or an unsound scope declaration, not a replication miscopy — fix the history table (or the declaration), then re-run. A warning notification fires when this first appears. Filter all such runs with display_status=scd_invariant_violated on the runs API. |
| Pass · Reduced timestamp fidelity | The run detail's Reduced timestamp fidelity card names each affected column, its side and the reader. Two readers do this today. A Delta TIMESTAMP read through Trino (timestamp(3) with time zone): Delta TIMESTAMP_NTZ columns are not affected, and for a microsecond-exact comparison you read the table through Databricks or Snowflake Delta Direct. A TIMESTAMP read through a Trino Hive catalog at its default hive.timestamp-precision (MILLISECONDS, shown as timestamp(3)): set hive.timestamp-precision=MICROSECONDS in the Trino catalog properties, the column then reads as timestamp(6) and later runs are not marked. Stored microseconds can also make a run against a faithful source fail on that column. Filter all such runs with display_status=reduced_fidelity on the runs API. |
| Readiness hold | The run detail says what readiness was waiting on, what the platform does next, and what to check (the replication job, the load pipeline's marker, the connection), above the full evaluation trail. The next run checks again. A Readiness hold notification rule tells you when one happens. To hear about it while the run is still waiting, add the Replication latency - still holding trigger to a rule and set its Early readiness-hold alert after threshold (30 minutes by default): you get one early warning per hold episode, then an all-clear when readiness passes again. Consecutive held runs of the same mapping share one warning. The card also has a Replication tool line, on every run (held or not): whether the replication tool was asked and what its task reported (for example "reported its task RUNNING" - meaning the hold is the watermark / latency check, not the tool), or why it could not be confirmed and whether readiness fell back to the watermarks (advisory) or stopped (strict), or that it was not consulted (integration disabled, no adapter for that tool, no replication profile), or that no probe was recorded for the run. When the tool's task is running, the probe also asks the tool for its own replication lag. If the tool reports one, the line shows it labelled as the tool's figure, next to the platform's own watermark lag check (for example "The tool reports 42s of replication lag (its own figure; our watermark lag check: 12s)"): the two are measured differently and one never stands in for the other. Debezium reports a figure only when its profile sets the optional Metrics endpoint (the JMX exporter on the Kafka Connect worker - see Replication profiles); a connector that has not streamed an event since it started has no figure yet. If the tool reports no figure, or the request fails or times out, the line says "The tool did not report a lag figure", with the reason when there is one - never a zero - and readiness carries on exactly as it would have. No replication tool reports an in-flight estimate, and the line says so. A platform_admin can re-trigger with Bypass readiness check (audited). |
| Config refusal | The run detail names the family and the refused setting (the error message, VPC-local). Fix what it names - restore or re-point the object, correct the watermark / filter / partition configuration, or fix the duplicate keys upstream - then trigger a new run; retrying without a change gives the same refusal. A Config refusal notification rule tells you when one happens (a warning, not a page). Filter all such runs with display_status=config_refusal on the runs API. |
| System problem | There is no retry button; trigger a new run. If it recurs, this is a platform issue, not a data issue. |
| No new data · Open issues | Nothing to do on this run. Drill into the mapping to see the outstanding set. |
| Running · Throttled / Queued | Cancel if you need the capacity elsewhere. |
Reading the run detail¶
Cards appear only when relevant, so a clean run is short and a failed run is long.
- Data Window — the mode, the window bounds, row counts per side, and the observed watermarks. If you are ever unsure what was compared, this is the card that answers it.
- Proof Phase — the two hash totals and whether they matched. Note that "Proof DB time (both sides, run in parallel)" sums both engines and can legitimately exceed the run's wall-clock duration.
- Compare Phase — mismatched rows and "wrong values" (cells).
- Engine Decisions - what the engine decided for this run and in what order (see below).
- Prior Open Issues Rechecked — keys re-checked, resolved this run, still failing. This is how you see a backlog shrinking. Only keys that were already open when this run started count here; a run that finds a break for the first time — including a break that had been fixed and has now recurred — shows no panel, because there was nothing prior to recheck.
- Current mapping status: N open issues — deliberately live as of now, not part of this run's record. Do not read it as this run's output.
- Reduced Precision by Policy — lists any field compared at a relaxed precision, so a reviewer can see where exactness was traded away.
- Data Volume and Prior Audit Bundles - data volume and evidence-export history. Under Data Volume, a one-line Resources used summary appears only when a provider-metered unit (credits, DBUs, billed TiB or slot-hours) has a value for the run; it links to the full ledger in Evidence > Process steps > Resources used (see Reconciliation Consumption). The run detail page has no consumption card.
Engine Decisions¶
One row per decision the engine took for this run, in order: the readiness check (ready, held, and why), the comparison window chosen and how long it spans, the schema drift check, the row-count and fingerprint comparison (including whether it ran inside the database or fell back to the reference path), the partition plan and each partition's result, how many mismatched rows were found, any diff limit that applied, each replication-lag recheck, and how the run finished. It is the quickest way to answer "why did this run end the way it did?" without reading logs.
Each row shows a time, the phase, a plain-English label and a few short facts (counts, yes/no flags, short codes). It never shows a table, schema or column name, a data value, a watermark value or a hash - those are refused before the row is written - so every role that can open a run (auditor and up) sees it. The panel shows up to 200 decisions at a time with Load more for the rest; a run records at most 500 (a final row says so if the cap was reached). Runs that started before this log existed show none. The same log is included in a Diagnostic Bundle, and retention removes it with the run's summary evidence.
Two counting conventions worth knowing¶
- "Missing in Src" means rows present only in the target. Counter-intuitive but correct — the row is missing from the source.
- Net Count Delta (Target) is a net figure. −475 does not necessarily mean 475 missing rows; it can be 500 missing and 25 extra. The UI decomposes it when a breakout is available.
Filters¶
Mapping · Mapping Group (hidden when you have no groups) · Status · Drift · date range · rows per page (25/50/100/200).
The Status filter uses the display vocabulary: Pass, Issue detected, No new data, Running, Readiness hold, Config refusal, Schema drift, System problem, Cancelled, Skipped.
The list's free-text search is deliberately hidden: it would only filter the page you are on rather than the whole result set.
The list polls every 10 seconds while any visible run is still going, and stops polling once everything is terminal.
Triggering a run¶
Trigger Run (needs mapping.trigger_run, or the operator role) offers:
- Window Scope — Incremental (watermark-derived), Full-table (whole
dataset; carries a cost warning), or Explicit window (you pin start and end;
entered in local time, converted to UTC). Full-table still waits for
readiness. It changes how much data is compared, not whether it is safe to
compare — so a full sweep is held, like any other run, when replication is
behind. To compare regardless, a
platform_adminuses Bypass readiness check, which is audited. - Comparison Depth — Full reconciliation (proof → keys → field diff), or Proof-only (aggregate hash, no field evidence, lower cost).
- Priority — low 25 / normal 50 / high 75 / critical 100. Higher runs first when the platform is congested.
- Bypass readiness check —
platform_adminonly, audited, with a live warning when ticked. - Notes — optional, recorded in the audit log, and shown on the run.
Only enabled mappings can be triggered.
Cancelling¶
Cancel appears on in-flight runs, in both the list and the detail page, for
every state the API accepts — including executing.
For a dispatched or executing run the workflow engine is cancelled first; if that fails the API returns an error and the database is left untouched, so you never get a run marked cancelled that is secretly still running.