Re-running and Restarting Jobs

Three ways to recover from a failed (or otherwise finished) job without retyping input: Re-run, which starts a brand-new job from the beginning; Restart from a step, which reruns only the steps affected by a chosen step and carries the rest forward unchanged; and Restart failed steps, which does the same for every failed or cancelled step at once.

All three are available from the job detail page once a job reaches a terminal state (completed, failed, or cancelled), and only on top-level jobs — a job you started yourself, or one a trigger, webhook, schedule or retry started. A child job of a type: task step, an agent tool call, or a hook job cannot be re-run or restarted: both actions create a brand-new job with no parent, so the original parent would never learn about it. Restart or re-run the top-level job instead. The buttons are hidden on those jobs, and the API rejects them with 400.

Re-run

The Re-run button on a job’s detail page opens the task’s execute form prefilled with that job’s original input (raw_input), including connection selections and non-default secret fields. Adjust anything you like, then submit — it creates a normal new job via POST /api/workspaces/{ws}/tasks/{name}/execute, running every step from scratch on the current workspace revision.

The new job records source_type: "rerun" and source_job_id pointing at the job it was re-run from, so the job detail page shows re-run of <id> under the job title (and in the Source card), linking back to the source job. A restarted job shows restart of <id> from <step> the same way (restart of <id> of failed steps <a, b, …> for a restart of failed steps). Re-run is only offered for jobs whose raw_input was captured — jobs created before this feature (or via legacy clients) show “This job predates Re-run prefill” and fall back to blank defaults.

The link goes the other way too. A job that was restarted or re-run shows a Later runs card on its detail page, with one row for each new job, newest first: the job id (a link to it), its status and, for a restart, the step it began at (restart 0438a27a failed from ai_sources) or the failed steps it began at (restart 0438a27a running failed steps ai_maintain, ml-prediction-master). The card shows the five newest runs and counts the others (+2 more). The page keeps the card current while it is open: a new run, or a run whose status changes, shows within 30 seconds (10 seconds while one of the runs is still running). Jobs that the job’s hooks created are not listed, because they are not new runs; each of them links back with hook for <id>. A task retry has its own Retried link. With ACL configured, the card leaves out a run that you may not view: the restart of a pinned job takes its folder from the commit it ran at, which can differ from the source’s folder.

A json field starts a re-run in replay mode, masked or not: the previous run’s value is reused exactly (the server copies the source job’s stored value). Choose Edit to change it. A value typed in the form is parsed by the browser, so integers beyond 2^53 lose precision there — use the API or replay for those. API clients do the same with replay_fields: {"source_job_id": "…", "replay_fields": ["cfg"], "input": {…}}. A re-run must use a source job of the same task; replay_fields needs source_job_id, may only name the task’s fields, may not repeat a field given in input, and fails when the source has no value for a required field with no default — each a 400.

Use Re-run when you want to change input, or when the failure is unrelated to which steps already succeeded. To skip re-running steps that already succeeded, use Restart from a step instead.

Restart from a step

Restart from a step creates a new job that reruns only a chosen step and everything downstream of it, while every other step is copied forward from the source job exactly as it ended. It uses the same input as the source job (no form) and the current workspace revision. If the task’s input schema has gained a required field with no default since the source job ran, the replayed input no longer satisfies it and the restart is rejected with 400 — use Re-run instead, which gives you the form.

Where to find it

• On a normal step’s detail panel, a Restart from here button appears once the job is terminal (next to the approval card, if any).
• For a for_each loop, the button is on the loop’s group header instead — individual loop instances are always restarted through the placeholder, never on their own.
• The button only appears if you have Run permission on the task (the same permission Execute requires).

Clicking it opens a confirmation dialog built from a dry-run preview (POST /api/jobs/{id}/restart with dry_run: true), so the counts and step names always reflect the current flow, not the flow the source job ran:

Restart {task} from {step}? Reruns N step(s): step-a, step-b, …. M step(s) are carried over unchanged.

The preview is computed when you open the dialog and the create request recomputes it, so a workspace reload in between can change what actually runs. The restart_steps in the create response is the authoritative list.

If any carried-over step ended failed and that step’s own continue_on_failure (under the current flow) doesn’t catch it, the dialog adds a warning that the new job will still end failed, and suggests restarting from an earlier step to rerun those failures too.

If a carried-over step ended failed but its own continue_on_failure lets the job pass, the dialog shows a note with those steps: they are copied, not rerun. Use Restart failed steps to rerun them too.

What gets rerun vs. carried over

Given the step you chose:

• The chosen step, plus any step that has been added to the flow since the source job ran (it has no matching row in the source job), form the initial set of roots.
• Every step that transitively depends on any of those roots is also rerun.
• Everything else is carried over: its new row is stamped with the source job’s ending status, output, and error message verbatim, without actually running again. A carried step that was still in progress when the source job was cancelled comes over as cancelled.
• Steps that existed in the source job but have since been removed from the flow are dropped, same as Re-run.
• Automatic inclusion is keyed on step names: a step counts as “added” only if its name has no matching row in the source job. A step that was removed and later re-added under the same name still matches, so it is carried over, not rerun — restart from it explicitly if its definition changed. When a genuinely new name sits upstream of a step you were keeping, that downstream step is pulled back into the rerun set, so you can’t strand a carried step behind a step that has no matching prior run.

Carried-over steps show a muted carried over badge in the step timeline. Their detail panel doesn’t fetch logs for the new job — it links back to the source job’s logs instead, since nothing executed here.

Job settlement

The new job runs through the normal creation, promotion, and dispatch pipeline for its rerun steps. If the rerun set collapses immediately (for example every rerun step is unreachable because a carried dependency is failed and not tolerated), the job settles right away, and hooks and metrics still fire exactly once for that outcome — same as any other job that finishes at creation time.

A carried failed step whose own continue_on_failure (under the current flow) catches it does not fail the job. A carried cancelled step with no failures ends the job cancelled, not completed (skipped rows never decide job status).

Restart failed steps

Restart failed steps creates one new job that reruns every step that failed or was cancelled in the source job, and every step below them. Everything else is carried over, exactly as for Restart from a step: same input, current workspace revision, same rules for steps added to or removed from the flow, same limitations.

Use it when several independent branches failed for one cause — for example two workers were replaced at the same time and steps on four branches failed. Restart from a step starts at one step only, so the other failures would be copied as failed and their downstream steps skipped again.

Where to find it

The Restart failed steps button is in the job header, next to Re-run. It is shown when the job is terminal and top-level, you have Run permission on the task, and at least one top-level step failed, was cancelled or did not finish. Clicking it opens the same confirmation dialog, built from a dry run (POST /api/jobs/{id}/restart with {"all_failed": true, "dry_run": true}):

Restart the failed steps of {task}? Starts at N failed or cancelled step(s): step-a, step-b. Reruns M step(s): … K step(s) are carried over unchanged.

Which steps it starts at

• Every top-level step whose status in the source job is failed or cancelled. A step with continue_on_failure: true is included: its failure did not fail the job, but it is still a failure you may want to rerun.
• A step that did not finish (for example it was still running when the job was cancelled) is included too: Restart from a step would carry it as cancelled.
• A for_each loop is restarted as a whole when its placeholder ended failed; single instances are never restarted on their own. A loop whose instances were all cancelled (none failed) ends completed, so it is carried over, not restarted.
• A failed or cancelled step that is no longer in the task’s current flow is ignored.
• skipped steps are rerun only when they are below a start step (for example the steps skipped unreachable below a failed step).

If no step qualifies, the request is rejected with 400 No failed or cancelled steps to restart. The new job records the start steps in restart_from_steps (and restart_from_step is null).

Rollout

The button needs a server that knows all_failed. During a rolling upgrade, a request that reaches a replica still on the previous release fails with 422; try again when every replica runs the new release. The release adds migration 055, which only adds a nullable column (no table scan, no manual step). As with every migration, a server binary older than the migration refuses to start against the migrated database. To roll back, stop every new replica, run DELETE FROM _sqlx_migrations WHERE version = 55, then start the old release; the column can stay. A later upgrade applies 055 again without error.

Files

A task with file input fields keeps its files on both paths.

• Re-run: the form shows the file of the source job (name and size) with a Replace button. If you keep it, the new job uses the same file. If you replace it, the browser uploads the new file with source_job_id, so the permission check uses the task of the source job (also for a pinned job). The replacement is a normal upload and follows the upload lifetime.
• Restart from a step and Restart failed steps: the restart takes no new input. It uses the files of the source job.

Limitations

Restart from a step does not fully reconstruct the source job’s environment for carried-over steps:

• State snapshots ({{ state.* }}) resolve to the latest snapshot for the task at claim time, not whatever snapshot existed when the source job ran.
• Artifacts are not copied. A rerun step that reads /artifacts/ from a step that was carried over (rather than actually rerun) finds nothing there, because artifacts belong to the job that produced them.
• Revision drift: carried-over output was produced under the source job’s workspace revision, but the rerun steps run under the current one. If templates or scripts changed incompatibly, a rerun step consuming a carried step’s output can fail at render or run time.
• Cross-workspace carried steps keep the current action definition (in case it changed) but the old output — the only link back to what actually produced that output is the job’s source_job_id.
• Secret redaction on carried output and error messages only covers currently configured connection/secret values, same limitation as job input generally — a value from a since-removed or rotated connection, or a user-typed secret, is not redacted. The one exception is a value that came from a git ref: the restart is masked with the secrets of every pinned commit the source job was connected to as well.
• Child jobs created by a carried type: task step are not re-linked to the new job; only the step’s final output comes along.
• Duration statistics: restart jobs (whole job and per-step) are excluded from the task duration stats used for percentile charts and ETA, even when the restart reran from the very first step. A restarted job also never falls back to the whole-task p50 for its ETA — it either computes a step-weighted estimate from steps it’s actually rerunning, or shows none.

If a failure you want to retry isn’t in the set you’d get by restarting from where it happened (because an earlier carried step is also failed and untolerated), restart from that earlier step instead so it reruns too — or use Restart failed steps, which reruns every failure in one job.

Pinned jobs

Re-running or restarting a job that runs on a git ref (for example, one started by a trigger with ref:) re-resolves that ref: a branch moves to its current tip, a tag or a commit SHA stays where it is. The task is looked up at the new commit, so it may exist only on that branch, and the new job runs pinned to that commit with the same ref. A restart’s plan is computed against the flow at the new commit; carried-over steps keep the output they produced at the old one.

Access is checked against the folder the source job’s task declared at its commit, and then against the folder the task declares at the re-resolved commit (the folder the new job carries). Both actions need Run on both (no access → 404, View → 403), so moving a task into a stricter folder on its branch also restricts who can re-run or restart its older jobs. A re-run must be posted to the source job’s own task name (another name is a 400). If the task no longer exists at the re-resolved commit, or the ref itself is gone, both answer 400; if the git server is unreachable, 500 — the restart preview contacts the git server too.

The new job’s input and output are masked with the secrets of every pinned commit its source job was connected to, as well as its own: a re-run replays the source’s input, and a task retry’s input is the failed job’s fully resolved input, so a re-run of a retry can carry a secret of the older commit.

The UI hides Re-run and Restart for a pinned job whose task does not exist on the default branch, because the task page reads the default branch. Use the API for those: POST /api/workspaces/{ws}/tasks/{task}/execute with source_job_id (and the source’s raw_input as input), or POST /api/jobs/{id}/restart.

Related

• Hooks — workspace-level hooks fire for both rerun and restart jobs (they’re top-level sources), and hook.failed_steps[].carried_over flags failures that came from the source job rather than this run.
• Retry Mechanisms — automatic step/task retry, a different mechanism for handling failures without user intervention.
• API Reference — POST /api/jobs/{id}/restart request/response shapes.