Recovery

If a worker dies mid-step (crash, OOM, network partition), or a step sits in ready state with no worker able to claim it, the server automatically detects the problem and recovers stuck jobs.

How it works

The recovery sweeper runs on a configurable interval (default: 60s) and performs these phases:

1. Stale worker detection: Workers whose last heartbeat exceeds the timeout are marked inactive. Running steps assigned to inactive workers are failed.
2. Step timeout enforcement: Running steps that have exceeded their configured timeout are failed.
3. Job timeout enforcement: Running jobs that have exceeded their configured timeout are cancelled.
4. Unmatched step detection: Steps stuck in ready state beyond unmatched_step_timeout_secs are checked against active workers. If no active worker has the required capability and tags to claim the step, it is failed with a clear error message.
5. Stalled pinned jobs: A running job on a git ref with no step ready, claimed, running or suspended is moved forward again. This picks up a job that could not continue because its commit was briefly unavailable (for example, a step finished on a replica that had not loaded the commit, during a git outage). If the commit can never load again, the job is failed. A job log line is written on every attempt while the commit stays unavailable.

Phases 1 and 2 fail a running step only while it is still the run they observed (the same worker and start time). A step that finished, or that was released and claimed again in the meantime, is left alone.

After each failure, the orchestrator cascades: dependent steps are skipped, the job is marked failed, and parent jobs are notified.

Configuration

Add an optional recovery section to server-config.yaml:

recovery:
  heartbeat_timeout_secs: 120        # Seconds without heartbeat before stale (default: 120)
  sweep_interval_secs: 60            # How often the sweeper runs (default: 60)
  unmatched_step_timeout_secs: 30    # Seconds a ready step waits before checking for matching workers (default: 30)

When the recovery section is omitted, recovery runs with defaults. There is no way to disable it — it’s always active.

In a multi-replica deployment the phases above run on the leader only. Each tick also evicts pinned commits that no active job needs from the replica’s own pin store; that runs on every replica, because the store is local.

The default heartbeat timeout of 120 seconds means a worker must miss 4 consecutive heartbeats (sent every 30s) before being considered stale.

The unmatched_step_timeout_secs grace period prevents false positives when workers are temporarily restarting or scaling up. A step is only failed if it has been ready for longer than this timeout and no active worker matches on both routing dimensions.

Capability + tag matching (Phase 4, post-041)

Migration 041 split the pre-041 single tags axis into two:

• required_ability on each step (derived from action type + runner). One of: "script", "docker", "kubernetes", "agent". Empty for task/approval (server-dispatched, excluded from Phase 4).
• required_tags on each step — only the user-declared action tags. No longer contains an ability token prefix.

Action	Runner	Required ability
script	local (default)	script
script	docker	docker
script	pod	kubernetes
docker	—	docker
pod	—	kubernetes
agent	—	agent
task, approval	—	— (server-dispatched, excluded)

Workers declare capabilities and (optional) affinity tags in worker-config.yaml:

capabilities:                # runners this worker supports (required)
  - script
  - docker
  - kubernetes
tags: []      # empty = no affinity axis; add labels to route matching steps
              # to this worker.
exclusive: false             # set true to refuse steps that don't request
                             # all of this worker's tags — the "reserved
                             # worker" pattern.

Phase 4 uses three SQL clauses in AND (post-042 affinity semantics):

worker.capabilities @> to_jsonb(step.required_ability)          -- ability matches
AND step.required_tags <@ worker.tags                            -- affinity: worker offers everything step wants
AND (NOT worker.exclusive OR worker.tags <@ step.required_tags)  -- reservation clause

That is, worker’s capabilities must contain the step’s ability, AND the step’s tags must be a subset of the worker’s tags. If the worker is exclusive, the worker’s tags must additionally be a subset of the step’s tags (i.e., set equality). If no active worker satisfies these after unmatched_step_timeout_secs, the step is failed with:

No active worker with required capability/tags to run this step

task/agent/approval/event_source steps are excluded from Phase 4 — they are dispatched server-side, not claimed by workers.

Recovery strategy

When a worker dies mid-step or a step has no matching worker, the step is failed, not retried:

• The step may have partially executed (side effects, partial writes)
• Retrying non-idempotent steps could cause data corruption
• Users can re-trigger the task manually or via on_error hooks

Worker reactivation

If a worker comes back online after being marked inactive, it is automatically reactivated on its next heartbeat. It can then claim new steps normally.

Data retention

Data retention is configured separately from recovery, in its own retention section. See Retention for details.

Recovery visibility

Recovery events are logged as server events on affected jobs, visible in:

• The “Server Events” panel on the job detail page in the UI
• The _server step logs via API: GET /api/jobs/{id}/steps/_server/logs