Troubleshooting¶

This page starts from what you see and works back to what to check. Most of what looks like a problem is the product refusing to give a verdict it cannot stand behind: a held run, a refused configuration, a schema it could not verify. Those are the platform working. The sections below say how to tell that apart from a real fault, and what to put in a report when it is one.

Every error the API returns carries a short error reference (the request id). It appears on the error message in the application and in the X-Request-ID response header. Quote it in any report: it is how support finds the server-side log line without you sending logs. See Report a problem for the Diagnostic Bundle, which collects the references, the run's decisions and the fleet state for you.

Start here - symptom to section¶

What you see	Most likely	Go to
A run reads Readiness hold · reason	Replication or the load pipeline had not finished; the run was deferred, not failed.	Readiness holds
A run reads Config refusal · family	The mapping cannot be compared as configured; a retry gives the same answer.	Config refusals
Schema drift, Schema unverified, Schema not verifiable	The live schema moved, or one side could not be introspected.	Schema drift and unverified schemas
Issue detected right after Running · Rechecking in-flight rows	Replication was still catching up when the rechecks ran out.	Rechecks ran out
The Engine Decisions log says the fingerprint ran on the reference path	A column type, transform or engine pairing cannot be hashed inside the database.	Reference path instead of pushdown
Diagnostics shows a worker stale, a version mismatch, or no worker serving a queue	A worker is down, mid-deploy, or running an older build.	Workers and versions
Runs sit at Running · Queued, · Throttled or · Checking readiness and never move	No worker is polling the queue, or the system's concurrency cap is reached.	Workers and versions
An API call answers 409 or 422 mentioning the Idempotency-Key	The same key was reused, or the first request is still being processed.	Idempotency conflicts
You land on the login page with a session expired notice	The access token expired or no longer validates.	Session expiry
Test Connection fails	Credentials pointer, network, or transport encryption.	Connection tests
A schedule is enabled but no run appears	The scheduler behind it is not confirmed running, or the schedule is outside its active window.	Schedules that do not fire
Evidence shows [masked] instead of values	A masking policy, by design.	Masked values
Something went wrong page, or a System problem · System error run	A platform or connector fault, not a statement about your data.	Platform faults

Readiness holds¶

Symptom. A run ends as Readiness hold with a reason: Replication latency, Replication stalled, Replication not running, Iceberg metadata lag, Load incomplete, Source unavailable, Replication status unknown or Configuration issue. Older runs may show the generic Source not ready.

Cause. Readiness checked the replication tool and the watermarks on both sides and could not find a window that both sides had finished delivering. The run was not compared. Comparing would have produced false mismatches.

What to check. Open the run. The Readiness card says what it was waiting on, what happens next, and what to look at, above the full evaluation trail. The reason names the direction:

Replication latency and Replication stalled: the replication job. The second one means the target did not move between checks at all.
Replication not running: the tool reported its task paused, stopped or failed. See Replication profiles.
Load incomplete: the load pipeline has not written its completion marker, or the target was still changing within the quiescence window.
Source unavailable: the platform could not read how far the data had loaded. Start with Test Connection on both systems.
Replication status unknown: strict CDC mode and the tool could not be reached. Either restore the tool's endpoint or relax the integration mode on the mapping if advisory behaviour is acceptable.
Configuration issue: the readiness settings themselves are invalid. Retrying will not help until they are corrected.

The two settings most often behind a recurring hold are max_lag_seconds (how far behind replication may be) and quiescence_seconds (how long the watermark must be stable). Both are in the configuration reference.

When to report. A hold that names Source unavailable while Test Connection succeeds on both systems, or a hold whose evaluation trail shows no evaluations at all. A hold with a clear reason and a replication job that is genuinely behind is not a platform issue. The full status vocabulary is in Runs.

Config refusals¶

Symptom. A run ends as Config refusal with a family: Object or connection config, Watermark config, Filter SQL, Duplicate keys, Partition strategy, Snapshot pin unavailable, Schema unavailable, Engine requirement, Data encoding, or Operator action required when no family applies yet.

Cause. After readiness passed, an activity hit a condition that no retry can change: a table or column that no longer exists, a revoked grant, a watermark that cannot be executed on that side, a filter that fails the SQL safety allowlist, more than one row sharing a key on one side, a partition strategy that overlaps or leaves rows uncovered, a time-travel pin outside the side's retention, an engine below the supported floor, or text that is not valid UTF-8. The platform refused rather than compare against a guess.

What to check. The run detail names the family and the refused setting (the message is VPC-local and never leaves your deployment). Fix what it names, then trigger a new run. Retrying without a change produces the same refusal. A Config refusal notification rule tells you when one happens; it is a warning, never a page.

When to report. A refusal whose family is Operator action required and whose message you cannot act on, or a refusal that recurs after you corrected exactly what it named.

Schema drift and unverified schemas¶

Symptom. One of: Schema drift · Not compared, Schema drift · Available data passed / failed, Pass · Schema unverified, Pass · Schema not verifiable, Schema unverified · Not compared, or the separate Schema review pending badge.

Cause. Every run introspects both sides and compares the live columns with the schema the mapping accepted. An added column never halts (it opens a pending decision). A lost or narrowed reconciled column follows the mapping's drift policy. A lost key column always halts. A side whose schema could not be read at all is recorded as unverified, and such a run can never present as an unqualified clean pass, whatever the policy.

What to check.

Not compared: the Schema Drift card lists what changed. If the change was intended, Refresh Schema on the mapping re-introspects and accepts the live tables as the new baseline.
Available data passed / failed: a reconciled column was excluded this run. Decide whether to accept the new schema or restore the column.
Schema unverified: introspection failed unexpectedly on a side. Almost always catalog permissions. The run detail records which side and why; the amber clears on the next run that verifies.
Schema not verifiable: the side is on an engine that structurally cannot introspect (raw object storage, for example). This is a standing property, not an alarm.

The policy knobs and their meaning are in Mappings.

When to report. Schema unverified on an engine that can introspect, after you have confirmed the service account can read the catalog. A drift finding on a table nobody changed.

Rechecks ran out (lag retries exhausted)¶

Symptom. The run spent time at Running · Rechecking in-flight rows and still ended as Issue detected. In the Engine Decisions log, up to three rows read lag retry attempted, each with how many keys were rechecked, how many resolved and how many remained.

Cause. When the proof fails on a mapping that declares a replication tool, the platform does not trust the first answer: it waits and rechecks the mismatched rows, three times, after 30, 60 and 120 seconds. If every mismatch resolves, the run passes as Pass · Resolved on recheck. If any remain after the third recheck, the run reports the issue. The recheck count and waits are fixed; they are not configurable per mapping.

A mapping with no replication tool gets no recheck at all unless a side was read at a pinned instant; its Process Steps card shows the In-Flight Recheck step as Skipped, with no replication tool as the reason. That is deliberate: without a tool there is no lag to wait out.

What to check.

Read the three recheck rows in order. A remaining count that keeps shrinking means replication was slower than about three and a half minutes of total wait: the rows are arriving, just late. Issues carry forward and are rechecked at the start of the next run, where they normally resolve. Watch the mapping's open-issue count fall.
A remaining count that does not move across all three rechecks, and again on the next run, is a real break. Open the evidence.
A recheck row with the reason code too many changed rows means the recheck was skipped because the changed-row set was larger than the recheck is allowed to re-read; the run's verdict stands on the proof.
If rechecks are routinely needed, replication is running close to your max_lag_seconds. Raising the readiness tolerance holds the run a little longer and avoids the recheck phase entirely.

When to report. Rows that the evidence shows present on the source and absent on the target for longer than your replication tool's own lag figure says they should be. That is a replication problem to raise with the tool's owner, with the run id and the mismatch keys export attached. A run that rechecks but writes no recheck rows at all is a platform issue.

Reference path instead of pushdown¶

Symptom. The Engine Decisions row for the fingerprint comparison says the hash did not run inside the database (pushdown no, fallback used yes), the Key Extraction step in Process Steps says row hashes were computed on the platform's reference path, the run took noticeably longer than usual, or an anomaly reports key-extraction throughput below its floor.

Cause. By default the platform computes each side's fingerprint inside the engine and moves only the aggregate. When a side cannot do that faithfully it declares so, and the platform computes that side on its own reference implementation instead. The result is still exact; it is slower and moves more data. When one side falls back, the other side is recomputed the same way so the two fingerprints are comparable. Common reasons a side cannot push down:

The two engines do not share a hash method, so neither side can be compared natively.
A transform in the field's chain is one the engines do not fold identically: uppercase, lowercase, collation fold, date truncate or numeric quantize. The chain editor says valid but forcing the reference implementation when you add one. See Field configuration.
A binary or large-object column compared content-exact on Oracle, DB2 or Redshift, which cannot hex a large object in SQL.
A JSON or array column, which is compared on canonical JSON rather than the engine's own text.
A hash plan for binary data over a column the engine declares as text.
An engine below the native floor: DB2 before 11.1, SQL Server before 2019.

What to check. The field grid for transforms and for BLOB compare modes (the editor marks a chain that forces the reference path). The two systems' engine versions. Whether the slowdown coincides with a configuration change.

When to report. A fallback on a mapping with none of the reasons above, or a run that became much slower with no configuration change on either side. The run narrative and the Diagnostic Bundle carry the decision rows; attach the run id.

Workers and versions¶

Symptom. The Diagnostics page (Monitor, operator and up) shows a worker as stale, a worker whose version or build does not match the API, no worker serving this queue, or runs that sit at Running · Queued or Running · Throttled or Running · Checking readiness and never advance.

Cause. Each worker writes a heartbeat every fifteen seconds. A row older than three intervals is stale: the process may be up, but it is not doing work. A worker that was replaced (a restart, a new pod) leaves its previous row behind until the platform retires rows older than a day, so a stale row whose queue also has a live worker is history, not an outage. The version column compares each worker's build to the API's: mismatch means an older or newer build is still serving; unverified means a build commit is missing on one of them (a development build), which is not the same as matching.

What to check.

Stale, and the queue has no live worker: the worker for that queue is down or its event loop is wedged. Restart it. Work sent to the queue waits until a worker polls it.
Mismatch during a deployment: expected while pods roll. Wait one heartbeat cycle after the rollout completes.
Mismatch after the rollout: a worker is running an image that was not rebuilt or restarted with the upgrade. Restart that worker's deployment.
Queued or Throttled and not moving, workers healthy: the mapping's systems have reached their concurrency cap. The run starts when a slot frees. Cancel if you need the capacity elsewhere.

When to report. A worker that stays stale after a restart, or a queue with a live worker whose runs still never leave Queued. Generate a Diagnostic Bundle; its fleet section and queue statistics are what support asks for first.

Idempotency conflicts¶

Symptom. An API client receives one of:

Status	Code	Meaning
400	`MISSING_IDEMPOTENCY_KEY`	Every state-changing request needs an `Idempotency-Key` header.
409	`IDEMPOTENT_REQUEST_IN_PROGRESS`	The first request with this key is still running. Retry after the `Retry-After` interval (five seconds) to receive its result.
422	`IDEMPOTENCY_KEY_BODY_MISMATCH`	This key was already used with a different request body. Use a new key for a different operation.
409	`DUPLICATE_IDEMPOTENCY_KEY`	On triggering a run: the `idempotency_key` in the request body was already used for an execution request.

Cause. The key is claimed before the handler runs, so of several identical requests exactly one executes; the rest receive the cached response for twenty-four hours, or the in-progress answer while the first is still running. The run trigger additionally keeps its own key on the execution request, which is unique per mapping and window.

What to check. The application mints a fresh key for every click, so these errors come from scripts and integrations. Reuse a key only to retry the very same request; mint a new one for a new operation.

When to report. An in-progress answer that persists for minutes on a request you know has finished, or a 503 from a notification-channel or ticket test (the key store is unavailable, and for endpoints with an external side effect the platform refuses rather than risk sending twice).

Session expiry¶

Symptom. The application returns you to the login page with a session expired notice, sometimes in the middle of a page.

Cause. An API call answered 401 while you held a token. The token expired, or the identity provider's signing key rotated so the token no longer validates. The application clears the session and sends you to sign in rather than render an empty page that looks like missing data.

What to check. Sign in again. If you are signed out again within seconds of signing in, the problem is between the platform and the identity provider: check Authentication settings (platform or tenant admin) and the identity provider's clock and key rotation. The error reference on the notice goes in the report.

When to report. Repeated immediate sign-outs after a successful login, with the error reference and the time.

Connection tests¶

Symptom. Test Connection on a system fails.

Cause. The pointer to the secret does not resolve, the network path is closed, the credential is wrong, or the server cannot offer encrypted transport and the platform refused to connect unencrypted.

What to check. The failure text is the first line of the driver's message, at most 200 characters; a timeout reads timeout. The full message is in the server log under the error reference. A PostgreSQL or MySQL failure mentioning TLS or SSL is the fail-closed default at work; see Testing a connection for the choice it asks you to make.

When to report. A test that fails with no reason text at all, or a credential that works from another client on the same network and fails here.

Schedules that do not fire¶

Symptom. A schedule shows enabled, but no run appears at the expected time.

Cause. Saving a schedule also asks the workflow engine to start the durable scheduler behind it. If that second step failed, the schedule looks enabled with nothing behind it. Otherwise the schedule is simply outside its active window: days of week, daily start and end, effective dates, or a close-mode toggle that disabled its tag.

What to check. The Next Run column. A red Scheduler not running badge or an amber Syncing badge that persists is the first case; the platform retries every maintenance tick and the badge clears when the scheduler worker and Temporal are reachable. No badge means the scheduler is confirmed running and the schedule's own window explains the gap. See Is the scheduler actually running it?.

When to report. No badge, a window that includes now, and still no run after one full interval.

Masked values¶

Symptom. Evidence shows [masked] where you expected a value.

Cause. One of three policies: the mapping never stores raw diff values, a field is masked in the UI, or your role is not permitted to see that data classification. The first cannot be undone by anyone; the values were never recorded. See Masking.

When to report. Never on its own. If you believe the policy is wrong, that is a configuration conversation with the mapping's owner.

Platform faults¶

Symptom. A Something went wrong page in the application, an error message with a reference and no actionable text, or a run that ends as System problem · System error.

Cause. A platform or connector fault: a worker lost its database, a driver raised something unexpected, an evidence write failed after its retries. None of these is a statement about your data.

What to check. A system-error run has no retry button; trigger a new run. If it recurs, it is a platform issue. The Diagnostics page's recent API errors and run errors list the references.

When to report. Any recurrence. Use Report a Problem from the error or from Diagnostics and attach the Diagnostic Bundle to the support email: it carries the references, the run's engine decisions, the fleet state and the queue statistics, with names tokenised by default and no evidence values. See Report a problem.

What a good report contains¶

The error reference (request id) from the message or the response header, and the time.
The run id when a run is involved.
The Diagnostic Bundle, generated by an operator from Report a Problem.
What you expected, what you saw, and whether it recurs.

Support will not ask you for logs, credentials or data values. Nothing the bundle contains leaves your deployment except by the email you send.