Reliability: retry, failures, replay

What happens to an event after it is accepted, how many times the platform retries it, what each terminal state means, and how to replay the ones that did not land. The guarantee behind the whole page: a record that fails is never silently dropped. It always ends up as a row you can see, filter, and replay.

Every number below is read from the running platform (the sync worker, the queue definitions, the API handlers), not from a service-level aspiration.

The delivery path

An accepted event becomes one message on a FIFO queue (inbound-sync.fifo for events entering HubSpot, outbound-sync.fifo for events leaving it). The flow worker receives the message, runs every matching flow, and then either acknowledges it, leaves it for another attempt, or writes a terminal row and discards it.

Only the attempt count is fixed. The wait between attempts is configurable, and the platform lengthens it on its own in two cases (a 429 carrying Retry-After, and an association whose counterpart has not synced yet). Both are described below.

Which failures are retried

The engine classifies every failure into one of three classes, and the class decides whether a retry can possibly help:

A fatal outcome is rarely connector-wide (a wrong form GUID or a missing OAuth scope breaks one flow while its siblings keep working), so the breaker is per flow: after 3 consecutive fatal outcomes on the same (connector, flow) pair, that flow is skipped while every other flow of the connector keeps syncing and POST /api/v1/events keeps accepting. Only when every enabled flow has tripped does the connector itself move to error, which blocks ingestion with a 423.

Recovery is explicit: POST /api/connectors/{id}/flows/{flow_name}/breaker/reset closes one flow’s breaker, and setting the connector back to active (PATCH /api/connectors/{id}) reopens the connector gate and closes every flow breaker at once.

Backoff

Default: flat

Out of the box there is no per-attempt delay: a retried message simply waits out the queue’s 120 s visibility timeout. That is the right shape for a target that is briefly unavailable and wrong for a dependency that takes minutes to arrive, which is why the backoff is opt-in per connector.

Opt-in exponential backoff

Set the retry block inside the connector config’s settings object, then publish:

{
  "version": "3.0",
  "entities": [],
  "flows": [],
  "settings": {
    "retry": {
      "backoff_base_seconds": 60,
      "backoff_max_seconds": 600
    }
  }
}

With backoff_base_seconds: 60 the waits are 60 s, 120 s, 240 s, 480 s, capped by backoff_max_seconds, which stretches the same five attempts over roughly fifteen minutes instead of eight.

The attempt count is never widened by this block: settings.retry buys a longer wait, not more tries. The 5-attempt ceiling is the queue’s own redrive policy and a connector cannot exceed it.

settings is a free-form object with no sub-schema, so a mistyped value ("60s", a list, an object) does not break the connector: the worker logs a warning and falls back to the default for that key.

Retry-After on 429

When an outbound target answers 429, plugsync treats it as transient (rate limiting is a slowdown, not a broken event) and honors the response’s Retry-After header. Both RFC 7231 forms are accepted, delta-seconds (Retry-After: 120) and HTTP-date, and the parsed value is clamped to 12 hours (43200 s), the SQS visibility ceiling. A malformed header is ignored and the message falls back to the queue’s own backoff.

Retry-After is honored even without the opt-in settings.retry block. When both apply, the wait is max(exponential backoff, Retry-After): never sooner than the target asked, never shorter than an already-longer backoff.

Associations waiting on a counterpart

An associate_many step whose counterpart record has not been synced yet gets its own, longer schedule inside the same five attempts: 60 s, 120 s, 240 s, 300 s (capped), about twelve minutes of cumulative wait. If the counterpart never appears, the record goes terminal with a distinct association_endpoint_unresolved error code rather than a generic transient failure, so “the other side never arrived” is not confused with “the target was flaky”.

Event lifecycle: the terminal states

One sync_events row is written per record, per terminal outcome. The full set of statuses:

Two distinctions are worth internalizing, because they are the ones that decide what you do next:

• parked versus failed. parked means “give it more time and replay it”; failed means “this needs a fix”. A message whose flows produced a mix of outcomes (one flow waiting on a dependency, another genuinely broken) is reported as failed, never parked: a replay alone could not fix it, so the state does not promise that it could.
• partial_success versus parked. If any step already wrote to HubSpot, the row is partial_success even when the failure was transient. The write-completeness signal wins, because hiding a landed write would be worse than under-reporting the failure class.

There is no automatic re-drive once a dependency clears. A parked record stays parked until you replay it.

Replaying a failed record

Finding replayable records

GET /api/connectors/{connector_id}/sync-events returns the most recent records, optionally filtered by status. Every row carries a boolean replayable, so the dashboard (and your own tooling) can show a Replay action without a second round trip:

{
  "id": "3f2c...",
  "entity_name": "order",
  "status": "parked",
  "direction": "inbound",
  "external_id": "A-1001",
  "error_detail": "plugin requested retry: company not synced yet",
  "replayable": true
}

replayable is true only when all three conditions hold:

1. the status is failed, partial_success, parked, or quarantined;
2. the row carries a captured data_snapshot (rows predating replay support do not);
3. the direction is inbound or outbound, the two directions a worker queue actually consumes.

For a batch view, GET /api/connectors/{connector_id}/sync-events/summary returns a zero-filled count per status in one call, scopable with since and until.

Triggering the replay

curl -X POST \
  https://app.plugsync.com/api/connectors/{connector_id}/sync-events/{event_id}/replay

The replay re-enqueues the row’s stored data_snapshot onto the same queue the worker normally consumes from, so it gets the full retry, classification and heartbeat machinery rather than a second execution path. The response identifies the new run:

{ "status": "queued", "message_id": "abc...", "trace_id": "replay-3f2c...-9d1e4b7a" }

The replayed message is pinned to the config version that is active at replay time, which makes the obvious repair loop work as expected: fix the config, publish, replay. The replay never runs against the stale version that produced the original failure.

A quarantined record uses the same endpoint with ?force=true. That is a deliberate confirmation step: releasing a quarantine applies the event as-is and bypasses the entity’s identity_guard for that one replay, so it cannot happen by accident.

These endpoints authenticate as your dashboard user, not with the connector’s event API key.

Record mode: full payload archive and range replay

Per-record replay works from the data_snapshot captured on failure rows. Record mode is the broader option: it archives the raw payload of every event crossing the connector, successes included, in both directions. It is the substrate for parallel runs and for re-running a whole window of traffic.

It is opt-in and off by default. Enable it in the connector config and publish:

{ "settings": { "record_mode": true } }

Retention is an org-level setting (record_retention_days, set via PATCH /api/orgs/{id}). The archive expires objects on fixed lifecycle classes: 7, 30, 90, 180, 365 days, default 30. A requested value snaps up to the next class, so the archive never expires earlier than asked.

A range replay is paced to protect the target: 60 events per minute by default, 600 maximum (pace_events_per_minute). It can also be aimed at a different connector of the same org via target_connector_id, which is how a parallel run against a mirror connector is done without touching the live one.

Two failure modes worth anticipating: a 409 means record mode was never enabled on this connector and its archive is empty (enable it, publish, and let events flow first), and a 423 means the target connector is paused or in error, which would leave the replayed messages unconsumed.

Dead-letter queues

DLQs exist, and they are platform-managed rather than a customer surface.

The important part is what does not reach them. A classified application failure never ages into a DLQ: at budget exhaustion the worker writes the terminal failed / parked / partial_success row and then discards the message, so the failure shows up in your sync events, with its payload, ready to replay. The DLQ only catches messages that never reached a classified outcome at all, for example a worker-level fault repeating across every redelivery.

Operationally: a DLQ alarm fires at the first message (threshold >= 1, not a rate), DLQ retention is 14 days, and every operator inspect, replay or delete is audit-logged with actor and timestamp. The admin endpoints behind it are restricted to platform administrators and are not part of the public API surface.

Related reference

• Event API - the ingestion contract, its status codes, and per-event /status and /verify.
• Troubleshooting - symptom-first index of the failure modes you are most likely to hit.
• settings - where retry and record_mode live in the connector config.
• Loop prevention and conflicts - why a legitimate no-op is reported as skipped.
• Draft and publish - config changes, including settings, apply only once published.