# Strøm — Workflow YAML Reference

> Strøm is a workflow/task orchestration platform. Workflows are defined as YAML
> files in a `.workflows/` directory. This document contains everything needed
> to author valid workflow YAML: actions, tasks, triggers, hooks, templating,
> input/output, and secrets.
>
> Site: https://fremvaerk.github.io/stroem

## Quick Reference — Validation Rules

When generating Strøm workflow YAML, these constraints are enforced at parse time:

**Action types:**
- `type: script` — requires `script` or `source`. Cannot have `image`. Supports `language` field: `shell` (default), `python`, `javascript`, `typescript`, `go`. Optional `dependencies` (package list) and `interpreter` (override auto-detected binary).
- `type: docker` — requires `image`. Runs container as-is (no workspace mount).
- `type: pod` — requires `image`. Runs as Kubernetes pod (no workspace mount).
- `type: script` + `runner: docker` — scripts inside a Docker container with workspace at `/workspace`.
- `type: script` + `runner: pod` — scripts inside a Kubernetes pod with workspace.
- `type: task` — requires `task` field. Cannot have `cmd`, `script`, `image`, `runner`, or `language`.
- `type: agent` — requires `provider` and `prompt`. Calls an LLM (19 providers: anthropic, azure, cohere, deepseek, galadriel, gemini, groq, huggingface, hyperbolic, llamafile, mira, mistral, moonshot, ollama, openai, openrouter, perplexity, together, xai). Optional `system_prompt`, `output` (structured JSON schema), `model`, `temperature`, `max_tokens`.
- `type: approval` — optional `message` (Tera template). Pauses job for human approval, reuses Phase 5d infrastructure.
- `type: script` + `image` is **rejected** — use `type: docker` or `runner: docker` instead.

**Task actions:**
- `task` field must reference an existing task — in the same workspace, or another workspace's task as `ws.task`.
- Self-referencing tasks (direct or via hooks) are rejected. With `ref`, the same task at another commit is allowed (bounded by the nesting depth).
- Maximum nesting depth: 10 levels.

**Git refs (`ref:`):**
- At parse time (`stroem validate`): `ref` is allowed only on a flow step (qualifies `action`), a `type: task` action (qualifies `task`), and scheduler / webhook triggers (qualifies `task`) — not on any other action type, hooks, event-source triggers or agent `tools: [{task: …}]` entries. The value must be a branch or tag name, `refs/heads/<name>`, `refs/tags/<name>`, or a full 40-hex commit SHA; other `refs/…` namespaces and templated values (`{{`, `{%`) are rejected.
- Not checked at parse time — rejected (400 / trigger MISSED) when the job is created or the trigger fires: a short SHA (it parses as a branch name, then fails as "ref … not found" with a hint to use the full 40-character SHA), a branch / tag / commit that does not exist, a library item (`lib.item`) with `ref`, a `ref` whose owner is a folder (non-git) workspace, an unknown workspace, a name missing at that ref, and a flow step whose ref'd action is `type: agent`.
- The referenced name is looked up at the ref (it need not exist on the default branch); `stroem validate` skips that check with a warning.

**Flow steps:**
- `action` must reference an existing action.
- `depends_on` must reference existing steps within the same flow.
- No cycles allowed in the dependency graph.
- Step names must not contain `[` or `]` (reserved for for_each instances).
- `for_each`: Tera template string (renders to JSON array) or literal array. Max 10,000 items.
- `sequential: true`: run loop instances one at a time (default: parallel).
- Inside loop instances: `each.item` = current element, `each.index` = zero-based index.
- `continue_on_failure` on a step: if this step fails, is cancelled, or is skipped because something above it failed, the steps that depend on it still run, and the failure does not fail the job. It never makes the step itself run. `continue_when_skipped` on a step: if this step is skipped by its own `when`, an empty `for_each`, or because a step above it was skipped the same way, the steps that depend on it still run. A step runs only when **every** dependency lets it through (completed, or not completed but carrying the matching flag) — there is no automatic convergence, so an if/else merge needs `continue_when_skipped` on each branch step. A failure fails the job unless a `continue_on_failure` catches it — on the failing step or on every path below it.

**Triggers:**
- `type: scheduler` — `cron` must be a valid 5-field or 6-field cron expression.
- `type: webhook` — `name` must match `^[a-zA-Z0-9_-]+$` (URL-safe).
- `mode` must be `"sync"` or `"async"` (default: `"async"`).
- `timeout_secs` must be 1–300 (default: 30). Only meaningful in sync mode.
- `type: event_source` — long-running queue consumer. Requires `task` (consumer task) and `target_task` (target for emitted jobs). Consumer's execution defined in referenced task. Emitted JSON on stdout creates jobs for target task. `env:` provides environment overrides. `restart_policy`: `always` (default), `on_failure`, `never`. `backoff_secs` for exponential backoff. `max_in_flight` for backpressure.
- `force_refresh: true` (scheduler, webhook) — reload workspace from source before creating the job.
- Scheduler / webhook `task` may name another workspace's task (`ws.task`), optionally with `ref`; the job is created in that workspace.

**Hooks:**
- `on_success` / `on_error` hook action references must exist.
- Hook actions can be `type: task` (creates a child job).

**Templates:**
- Step names with hyphens (e.g., `say-hello`) become underscores in templates: `{{ say_hello.output.* }}`.
- Template context: `input.*`, `<step_name>.output.*`, `secret.*`, `state.*` (task state), `global_state.*` (workspace state), `job.revision`, `job.ref` (empty unless the job runs on a git ref).

**Pod manifest overrides:**
- `manifest` field is only valid on `type: pod` and `type: script` + `runner: pod`.
- Must be a JSON/YAML object (not array or scalar).

**Connections:**
- `connection_types` define property schemas (`string`, `integer`, `number`, `boolean`).
- `connections` are named instances referencing a type. Required fields must be present.
- Task input `type` can reference a connection type — the name is resolved to the full object at job creation.
- Connection values support Tera templates (e.g., `{{ 'ref+...' | vals }}`).

---

## Workflow Basics

Workflows in Strøm are defined as YAML files containing **actions** and **tasks**.

## File location

Workflow files go in the `.workflows/` directory of each workspace and must have a `.yaml` or `.yml` extension. The server loads all files from this directory on startup.

A single YAML file can contain multiple actions and tasks.

For the default folder workspace, files are at `workspace/.workflows/`. For git-sourced workspaces, the repository must contain a `.workflows/` directory at the root.

## YAML structure

```yaml
actions:
  <action-name>:
    name: Human-Readable Name    # optional display name
    description: What it does    # optional description
    type: script
    script: "..."
    input:
      <param-name>:
        type: string
        description: What this parameter is for  # optional
        required: true
      <param-name>: { type: string, default: "value" }

tasks:
  <task-name>:
    name: Human-Readable Name    # optional display name
    description: What this task does  # optional description
    mode: distributed
    folder: <optional-folder-path>
    input:
      <param-name>: { type: string, default: "value" }
    flow:
      # Reference a named action:
      <step-name>:
        action: <action-name>
        name: Human-Readable Step Name  # optional display name
        description: What this step does  # optional description
        depends_on: [<other-step>]
        input:
          <param>: "{{ input.param }}"
      # Or define the action inline:
      <step-name>:
        type: script
        script: "..."
        name: Human-Readable Step Name  # optional
        depends_on: [<other-step>]
```

Values like `{{ input.param }}` and `{{ <step>.output.<field> }}` are [Tera](https://keats.github.io/tera/) templates — Strøm's template engine. See [Templating](https://fremvaerk.github.io/stroem/guides/templating/) for the full reference.

## Actions

Actions are the smallest execution unit. Each action defines a command or script that runs on a worker. See [Action Types](https://fremvaerk.github.io/stroem/guides/action-types/) for all supported types.

Actions support optional `name` and `description` fields for human-readable labeling:

```yaml
actions:
  greet:
    name: Greet User
    description: Sends a greeting message to the specified user
    type: script
    script: "echo Hello {{ input.name }}"
    input:
      name:
        type: string
        description: The user's name
        required: true
```

## Tasks

Tasks compose actions into a DAG (directed acyclic graph) of steps.

### Basic task

Tasks support optional `name` and `description` fields, and flow steps can also have their own `name` and `description`:

```yaml
tasks:
  hello-world:
    name: Hello World
    description: A simple greeting task
    mode: distributed
    input:
      name: { type: string, default: "World" }
    flow:
      say-hello:
        action: greet
        name: Say Hello
        description: Greet the user by name
        input:
          name: "{{ input.name }}"
```

The `name` and `description` are displayed in the web UI. When omitted, the YAML key (e.g. `hello-world`, `say-hello`) is used as the display label.

### Inline actions

For simple, one-off steps you can define the action inline instead of referencing a named action:

```yaml
tasks:
  hello:
    flow:
      say-hi:
        type: script
        script: "echo Hello, World!"
```

This is equivalent to defining a separate action and referencing it:

```yaml
actions:
  greet:
    type: script
    script: "echo Hello, World!"

tasks:
  hello:
    flow:
      say-hi:
        action: greet
```

Inline steps support all action fields (`type`, `script`, `source`, `image`, `runner`, `env`, `tags`, etc.) plus all step fields (`depends_on`, `input`, `continue_on_failure`, `timeout`, [`when`](https://fremvaerk.github.io/stroem/guides/conditionals/)):

```yaml
actions:
  notify:
    type: script
    script: "curl -X POST $SLACK_WEBHOOK -d message=\"{{ input.message }}\""

tasks:
  deploy:
    input:
      env: { type: string, default: "staging" }
    flow:
      build:
        type: docker
        image: node:20
        command: ["npm", "run", "build"]
      deploy:
        type: script
        runner: docker
        script: "deploy.sh {{ input.env }}"
        depends_on: [build]
        input:
          env: "{{ input.env }}"
    on_error:
      - action: notify
        input:
          message: "Deploy failed: {{ hook.error_message }}"
```

Use inline actions for steps that are unique to a single task. Use named actions when the same action is shared across multiple tasks or steps.

### Step dependencies

Use `depends_on` to define ordering. Steps without dependencies run as soon as a worker claims them. Steps with dependencies wait until all listed steps complete.

```yaml
tasks:
  deploy-pipeline:
    mode: distributed
    input:
      env: { type: string, default: "staging" }
    flow:
      health-check:
        action: check-status
      deploy-app:
        action: deploy
        depends_on: [health-check]
        input:
          env: "{{ input.env }}"
      send-notification:
        action: notify
        depends_on: [deploy-app]
        input:
          env: "{{ input.env }}"
          status: "success"
```

This creates a linear pipeline: `health-check` → `deploy-app` → `send-notification`.

### Parallel execution

Steps without mutual dependencies run in parallel:

```yaml
flow:
  checkout:
    action: git-clone
  setup-db:
    action: init-database
  # checkout and setup-db run in parallel
  build:
    action: npm-build
    depends_on: [checkout]
  test:
    action: run-tests
    depends_on: [checkout, setup-db]
    # test waits for both checkout AND setup-db
```

### Timeouts

You can set timeouts at the step level and the task level. Timeouts accept a human-readable duration string (e.g. `"30s"`, `"5m"`, `"1h30m"`) or a plain integer (seconds).

**Step timeout** — kills a single step if it runs too long (max 24h):

```yaml
flow:
  build:
    action: build-app
    timeout: 10m
  deploy:
    action: deploy-k8s
    timeout: 15m
    depends_on: [build]
```

When a step times out, it is marked as `failed` with the error "Step timed out". Its dependents run only if their own `depends_on` edge to this step accepts `failed` (e.g. `accept: [completed, failed]` or `accept: any`) — otherwise they're skipped `unreachable`. `continue_on_failure` on the timed-out step itself is unrelated to that: it only decides whether the timeout fails the *job* (see [Conditionals](https://fremvaerk.github.io/stroem/guides/conditionals/)). The worker also enforces the timeout client-side by cancelling the running process.

**Task timeout** — cancels the entire job if it runs too long (max 7d):

```yaml
tasks:
  deploy:
    timeout: 30m
    flow:
      build:
        action: build-app
        timeout: 10m
      deploy:
        action: deploy-k8s
        depends_on: [build]
```

When a task times out, all running steps are cancelled and the job is marked as `cancelled`. Note that the job timeout clock starts when the job is created, not when execution begins — queue wait time counts against the timeout.

Both timeouts are enforced server-side by the recovery sweeper (periodic check) and, for step timeouts, also client-side by the worker process for immediate enforcement.

#### Server-level defaults

Operators can set fleet-wide defaults in `server-config.yaml` so a forgotten timeout never leaves a stuck pod running forever:

```yaml
# server-config.yaml
default_step_timeout: 30m   # applied to every step without an explicit `timeout`
default_job_timeout: 4h     # applied to every task without an explicit `timeout`
```

The default fills in at **job-creation time** — the resolved value is written to the DB row and visible via the API, so operators can see exactly which timeout each running job is bound by. An explicit `task.timeout` or `flow_step.timeout` always wins, so individual tasks can override the default in either direction. The same defaults apply uniformly to retries, hooks, and `type: task` sub-jobs. Changing the config affects jobs created after the change, not in-flight ones.

To opt one task out of the global default — i.e., to make it genuinely unbounded — set its `timeout` to the maximum (`24h` for a step, `7d` for a task). Omit `default_step_timeout` / `default_job_timeout` entirely (or set them to `null`) to retain the original behaviour where timeouts are opt-in only.

### Handling step failures

By default, when a step fails, all downstream steps that depend on it are automatically skipped `unreachable` unless they explicitly tolerate the failure. The job is marked as `failed` once all steps reach a terminal state, unless every failure is caught.

A step runs only when its whole `depends_on` tree is satisfied. Each entry names which of the dependency's outcomes — `completed`, `failed`, `cancelled`, `skipped`, `omitted` — satisfy that edge:

- A plain `depends_on: [a]` is sugar for `accept: [completed]` — today's default: the dependency must complete cleanly.
- `{step: a, accept: [completed, failed]}` also tolerates `a` failing.
- `{step: a, accept: any}` runs regardless of how `a` ends.

There is no automatic convergence: every edge of a merge needs its own `accept` — tolerating one dependency's failure doesn't automatically tolerate a sibling's skip. See the [Conditionals guide](https://fremvaerk.github.io/stroem/guides/conditionals/) for branching patterns, the merge/if-else pattern, and skip reasons.

Separately, `continue_on_failure` on a step is self-scoped: it only controls whether *that step's own* failure fails the job, with no effect on whether anything downstream runs:

```yaml
flow:
  deploy:
    action: deploy-app
    continue_on_failure: true   # deploy failing doesn't fail the job
  verify:
    action: verify-deploy
    depends_on:
      - step: deploy
        accept: [completed, failed]   # verify runs even if deploy failed
```

`accept` and `continue_on_failure` are fully decoupled: a dependent can run after an upstream failure (`accept`) independently of whether that failure still fails the job (`continue_on_failure`, on the upstream step). A step that isn't really part of the flow's own DAG — paging on-call, tearing down infrastructure the flow itself never touched — still belongs in an `on_error` (or `on_cancel`) hook instead, so it fires exactly once on any terminal outcome without wiring an `accept` edge to every step that might fail:

```yaml
tasks:
  deploy:
    flow:
      deploy:
        action: deploy-app
    on_error:
      - action: notify
        input:
          message: "Deploy failed: {{ hook.error_message }}"
```

See [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/) for the full `on_success` / `on_error` / `on_cancel` reference.

### Loops (for_each)

Use `for_each` to fan out a step across a list of items. Each item spawns its own step instance, running in parallel by default:

```yaml
flow:
  deploy:
    action: deploy-app
    for_each: ["us-east-1", "eu-west-1", "ap-south-1"]
    input:
      region: "{{ each.item }}"
```

This creates `deploy[0]`, `deploy[1]`, `deploy[2]` — one per region. Inside the step, use `each.item` for the current value and `each.index` for the position.

The list can also come from a template expression referencing task input or a previous step's output:

```yaml
flow:
  get-targets:
    type: script
    script: "echo 'OUTPUT: [\"svc-a\", \"svc-b\"]'"
  restart:
    action: restart-service
    depends_on: [get-targets]
    for_each: "{{ get_targets.output }}"
    input:
      service: "{{ each.item }}"
```

Add `sequential: true` to run instances one at a time instead of in parallel. Downstream steps that depend on a `for_each` step receive the aggregated outputs as an array.

See the [Loops guide](https://fremvaerk.github.io/stroem/guides/loops/) for sequential mode, error handling, sub-job fan-out, and combining loops with conditional steps.

### DAG visualization

The web UI provides an interactive graph view for step dependencies:

- **Job Detail page**: Toggle between "Timeline" and "Graph" views. The graph shows each step as a node with live status (color-coded borders, animated edges for running steps). Click a node to view step details.
- **Task Detail page**: Tasks with more than one step display a dependency graph above the step list.
- **Fullscreen**: The expand button in the graph's top-right corner opens the same graph in a viewport-sized overlay with a wider zoom range, useful for large workflows. Selecting a node in the overlay also selects it on the page, so the step timeline is already focused when you close the overlay (collapse button or `Esc`).

### Organizing tasks with folders

Tasks can be organized into folders using the optional `folder` property. The UI displays tasks in a collapsible folder tree when folders are present.

```yaml
tasks:
  deploy-staging:
    folder: deploy/staging
    flow:
      run:
        action: deploy-app

  deploy-production:
    folder: deploy/production
    flow:
      run:
        action: deploy-app
```

Use `/` to create nested folder hierarchies. Tasks without a `folder` property appear at the root level.

## Validation

Use the CLI to validate workflow files before deploying:

```bash
# Validate a single file
stroem validate workspace/.workflows/deploy.yaml

# Validate all files in a directory
stroem validate workspace/.workflows/
```

The validator checks:
- YAML syntax and structure
- Action type validity (`script` needs `script` or `source`; `docker`/`pod` need `image`; `task` needs `task`)
- Runner field validity (`local`, `docker`, `pod`)
- Flow steps reference existing actions
- Dependencies reference existing steps within the same flow
- No cycles in the dependency graph
- Step `when` conditions and `for_each` expressions, and agent `prompt` / `system_prompt` templates, have valid Tera 2 syntax — and use only filters, tests and functions that exist (unknown ones are rejected, even in an untaken branch; see the [Tera 2 upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-tera-2/))
- Step names don't contain `[` or `]` (reserved for loop instances)
- Trigger cron expressions are valid
- Hook action references exist

---

## Action Types

Actions are the smallest execution unit in Strøm. Each action defines what runs and how.

## Overview

| Type | Description |
|------|-------------|
| `script` | Runs scripts in a runner environment with workspace files. Supports `runner: local` (default), `docker`, or `pod`. |
| `docker` | Runs a Docker image as-is — no workspace mount. |
| `pod` | Runs an image as a Kubernetes pod — no workspace mount. |
| `task` | References another task, creating a child job. Dispatched server-side — workers never see them. |
| `agent` | Calls an LLM as a workflow step. Dispatched server-side — workers never see them. |
| `approval` | Pauses execution for human approval. Dispatched server-side — workers never see them. |

**`docker` / `pod`** actions require `image`. The image's default entrypoint/cmd runs unless overridden. Use this when you have a self-contained image (e.g., DB migrations, deploy tools).

**`script`** actions require `script` (inline code) or `source` (file path). Workspace files are available at `/workspace` (read-only). Use this for build/test/deploy scripts that need your source code. Supports multiple languages via the `language` field.

**NOTE:** `type: script` with an `image` field is **rejected** by validation. Use `type: docker` or `type: script` + `runner: docker` instead.
## Field validity reference

Quick reference showing which fields are valid for each action type:

| Field | `type: script` | `type: docker` | `type: pod` | `type: task` | `type: agent` | `type: approval` |
|-------|----------------|----------------|-------------|--------------|---------------|------------------|
| `script` | ✓ (inline) | - | - | - | - | - |
| `source` | ✓ (file path) | - | - | - | - | - |
| `cmd` | - | ✓ | ✓ | - | - | - |
| `entrypoint` | - | ✓ | ✓ | - | - | - |
| `command` | - | ✓ | ✓ | - | - | - |
| `image` | - | ✓ | ✓ | - | - | - |
| `runner` | ✓ (`local`/`docker`/`pod`) | - | - | - | - | - |
| `language` | ✓ (`shell`/`python`/`js`/`ts`/`go`) | - | - | - | - | - |
| `dependencies` | ✓ | - | - | - | - | - |
| `interpreter` | ✓ | - | - | - | - | - |
| `manifest` | ✓ (pod runner only) | - | ✓ | - | - | - |
| `task` | - | - | - | ✓ | - | - |
| `provider` | - | - | - | - | ✓ | - |
| `prompt` | - | - | - | - | ✓ | - |
| `system_prompt` | - | - | - | - | ✓ | - |
| `message` | - | - | - | - | - | ✓ |
| `model` | - | - | - | - | ✓ | - |
| `output` | - | - | - | - | ✓ | - |
| `temperature` | - | - | - | - | ✓ | - |
| `tags` | ✓ | ✓ | ✓ | - | - | - |
| `env` | ✓ | ✓ | ✓ | - | - | - |
| `timeout` | ✓ | ✓ | ✓ | - | ✓ | ✓ |
| `input` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

## Script actions

**NOTE:** An action `input` can declare `type: file` fields. The step gets a read-only path to the file, and every runner step of a job gets every file of the job. See [File Inputs](https://fremvaerk.github.io/stroem/guides/file-inputs/).
Script actions run commands or scripts in a runner environment. By default, scripts run as shell commands, but you can use the `language` field to write inline scripts in Python, JavaScript, TypeScript, or Go.

### Inline command

```yaml
actions:
  greet:
    type: script
    script: "echo Hello {{ input.name }}"
    input:
      name: { type: string, required: true }
```

### Script file

Scripts are relative to the workspace root:

```yaml
actions:
  deploy:
    type: script
    source: actions/deploy.sh
    input:
      env: { type: string, default: "staging" }
```

### CLI arguments

Pass arguments to scripts using the `args` field. Each element is a Tera template:

```yaml
actions:
  deploy:
    type: script
    source: actions/deploy.sh
    args:
      - "{{ input.environment }}"
      - "--region"
      - "{{ input.region }}"
    input:
      environment: { type: string }
      region: { type: string, default: "us-east-1" }
```

For shell scripts, args become positional parameters (`$1`, `$2`, ...). For other languages, they appear as standard command-line arguments (e.g. `sys.argv[1:]` in Python).

The `args` field is only valid on `type: script` actions. Docker and pod actions should use `cmd`, `command`, or `entrypoint` instead.

### Environment variables

Actions can declare environment variables. Values support templating:

```yaml
actions:
  deploy:
    type: script
    source: actions/deploy.sh
    env:
      DEPLOY_ENV: "{{ input.env }}"
      API_KEY: "{{ secret.api_key }}"
    input:
      env: { type: string }
```

### Multi-language scripts

Use the `language` field to write inline scripts in languages other than shell. When `language` is set, the `script` content is written to a temporary file and executed with the appropriate interpreter.

| Language | Value | Aliases | Toolchain preference |
|----------|-------|---------|---------------------|
| Shell | `shell` (default) | — | `bash > sh` |
| Python | `python` | — | `uv > python3 > python` |
| JavaScript | `javascript` | `js` | `bun > node` |
| TypeScript | `typescript` | `ts` | `bun > deno` |
| Go | `go` | — | `go run` |

**Python example:**

```yaml
actions:
  analyze-data:
    type: script
    language: python
    dependencies:
      - pandas
      - requests
    script: |
      import pandas as pd
      import requests

      resp = requests.get("https://api.example.com/data")
      df = pd.DataFrame(resp.json())
      print(f"Rows: {len(df)}")
      print(f'OUTPUT: {{"count": {len(df)}}}')
    input:
      url: { type: string }
```

**JavaScript example:**

```yaml
actions:
  fetch-status:
    type: script
    language: javascript
    script: |
      const resp = await fetch("https://api.example.com/status");
      const data = await resp.json();
      console.log(`Status: ${data.status}`);
      console.log(`OUTPUT: ${JSON.stringify(data)}`);
```

**TypeScript example:**

```yaml
actions:
  generate-report:
    type: script
    language: typescript
    dependencies:
      - zod
    script: |
      import { z } from "zod";
      const Schema = z.object({ name: z.string() });
      const result = Schema.parse({ name: "test" });
      console.log(result);
```

**Go example:**

```yaml
actions:
  compute:
    type: script
    language: go
    script: |
      package main

      import "fmt"

      func main() {
          fmt.Println("OUTPUT: {\"result\": 42}")
      }
```

### Dependencies

The `dependencies` field installs packages before running the script. It requires a `language` other than `shell`.

| Language | Install command |
|----------|----------------|
| Python | `uv pip install --system <deps>` or `pip install <deps>` |
| JavaScript | `bun install <deps>` or `npm install <deps>` |
| TypeScript | `bun install <deps>` or `npm install <deps>` |
| Go | Dependencies are resolved automatically by `go run` |

### Interpreter override

Use the `interpreter` field to override the auto-detected binary:

```yaml
actions:
  legacy-python:
    type: script
    language: python
    interpreter: python3.11
    script: |
      import sys
      print(f"Using Python {sys.version}")
```

## Docker actions

Runs a prepared Docker image as-is. No workspace files are mounted.

```yaml
actions:
  migrate-db:
    type: docker
    image: company/db-migrations:v3
    env:
      DB_URL: "{{ secret.db_url }}"
    # No cmd — image's default entrypoint runs

  deploy:
    type: docker
    image: company/deploy-tool:latest
    cmd: "deploy --env production"

  custom-entrypoint:
    type: docker
    image: company/tool:v2
    entrypoint: ["/app/run"]
    cmd: "--verbose --env staging"
```

## Kubernetes pod actions

Runs a prepared image as a Kubernetes pod. No workspace files are downloaded.

```yaml
actions:
  train-model:
    type: pod
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    cmd: "python /app/train.py --epochs 10"
    tags: ["gpu"]
```

## Script in Docker

Runs scripts in a Docker container with the workspace mounted at `/workspace` (read-only):

```yaml
actions:
  lint-python:
    type: script
    runner: docker
    script: "pip install ruff && ruff check /workspace"

  run-tests:
    type: script
    runner: docker
    script: "cd /workspace && npm ci && npm test"

  analyze:
    type: script
    runner: docker
    language: python
    dependencies: [pandas]
    script: |
      import json
      import pandas as pd
      df = pd.read_csv("/workspace/data.csv")
      print("OUTPUT: " + json.dumps({"rows": len(df)}))
```

### Runner image override

Workers can be configured with a default `runner_image` in their configuration. When a `type: script` action with `runner: docker` or `runner: pod` runs, the configured `runner_image` is used instead of the published `stroem-runner` image. If no `runner_image` is configured, the system falls back to the default image.

Set this in your worker's `worker-config.yaml`:

```yaml
runner_image: "ghcr.io/myorg/custom-runner:latest"
```

This allows you to pre-bake tools, dependencies, and configurations into a custom runner image tailored to your workflows.

## Script in Kubernetes

Runs scripts as a Kubernetes pod with the workspace downloaded via an init container, at the step's revision (see [Runners](https://fremvaerk.github.io/stroem/guides/runners/#kubernetes-runner)):

```yaml
actions:
  gpu-test:
    type: script
    runner: pod
    tags: ["gpu"]
    script: "python /workspace/test_gpu.py"
```

:::note Pending pod timeout
Pods that remain in `Pending` state for more than 10 minutes are automatically terminated and the step is marked as failed. This prevents jobs from hanging indefinitely when pods can't be scheduled (e.g., insufficient resources, node affinity failures, image pull errors). The error message includes the pod's status reason when available.
:::

### Pod names and labels

Both pod runners (`type: pod` and `runner: pod`) name the pod `stroem-{first 8 chars of job id}-{task}-{step}`. Task and step are lowercased, every other character becomes `-`, repeated dashes collapse to one, and leading/trailing dashes are dropped. A name longer than 63 characters is cut short and ends in an 8-character hash of the full job id, task and step. This keeps the instances of a long-named `for_each` step (`step[10]`, `step[11]`) apart, although it does not guarantee unique names.

By default a step pod carries these labels, which you can use with `kubectl -l`:

| Label | Value |
|---|---|
| `app` | `stroem-step` |
| `stroem.io/job-id` | the job id |
| `stroem.io/task` | the task name, in the same lowercased form as the pod name (left out if nothing remains) |
| `stroem.io/step` | the step name with characters Kubernetes does not allow in a label replaced by `-` (`deploy[0]` → `deploy-0`) |

Label values are capped at 63 characters. A `manifest` override's `metadata.labels` is merged over these defaults, so it can replace them. The exact step name is in the `stroem.io/step-name` annotation. Strøm always sets this annotation, replacing any value a manifest gives it.

## Task actions

Actions of `type: task` reference another task by name. When a step using a task action becomes ready, the server creates a child job that runs the referenced task's full flow.

```yaml
actions:
  run-cleanup:
    type: task
    task: cleanup-resources

tasks:
  cleanup-resources:
    mode: distributed
    input:
      env: { type: string }
    flow:
      cleanup:
        action: cleanup-resources-action
        input:
          env: "{{ input.env }}"

  deploy:
    mode: distributed
    input:
      env: { type: string, default: "staging" }
    flow:
      build:
        action: build-app
      cleanup:
        action: run-cleanup
        depends_on: [build]
        input:
          env: "{{ input.env }}"
```

When the `deploy` task's `cleanup` step becomes ready, the server creates a child job running `cleanup-resources`. The child completes its own flow, and the parent step is marked as completed.

### Rules for task actions

- Must have a `task` field referencing an existing task — in the same workspace, or, since 0.16.5, another workspace by `workspace.task` name (see [Calling tasks in other workspaces](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/#calling-tasks-in-other-workspaces))
- May have a `ref` field — a branch, tag or commit SHA of the task's owner workspace to run the task from (see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/)); `ref` is valid on no other action type
- Cannot have `script`, `source`, `image`, or `runner` fields
- No worker tags required — task steps are server-dispatched
- Self-referencing tasks are rejected at validation time (with `ref`, the same task at another commit is allowed)
- Maximum nesting depth of 10 levels prevents infinite recursion
- Input templates are rendered server-side before creating the child job

### Parent-child relationship

- Child jobs track their parent via `parent_job_id` and `parent_step_name`
- When a child completes, the parent step is marked completed with the child's output
- When a child fails, the parent step is marked as failed
- The parent's orchestrator runs after child completion, promoting downstream steps
- The job detail's step view lists every child job created for the step (`child_jobs`), newest first
- A child job's detail page links back under its title — **child of `<parent>` at step `<step>`** — and `GET /api/jobs/{id}` returns both `parent_job_id` and `parent_step_name`

### Behaviour changes in 0.16.5

Cross-workspace `type: task` actions (a `task:` field naming another
workspace's task, directly or through a cross-workspace `action:` wrapper)
changed several existing behaviours:

1. `action: platform.run-deploy` (a cross-workspace `type: task` action) now
   runs `platform`'s task. Before, it ran the **caller's** task of the same
   bare name if one existed, else failed.
2. A `type: task` action's own `input` defaults are taken from the action
   definition persisted on the step at job creation, not re-read at
   dispatch: editing the action no longer changes an in-flight job's child.
   Secrets the defaults reference are still read live at dispatch, and the
   referenced task definition is still resolved live.
3. Connection-typed fields on a `type: task` **action's** schema are no
   longer resolved against the action schema — at creation (literal
   pre-check) or at dispatch. Resolution happens once, against the
   **task's** schema. A field the action declares as a connection but the
   task declares as a primitive, or that is absent from the task's schema
   entirely, now arrives as the name string, not the resolved object.
4. On a cross-workspace call, a non-string value in a connection-typed task
   input is rejected. It is a `400` at submit only when the value is a
   literal supplied by the caller's flow step, the step has no `when`, and
   the caller is not the task's workspace; a templated value, a value on a
   `when`-guarded step, or a foreign action default fails the step at
   dispatch if the step is reached. Within one workspace an object is still
   accepted as before.
5. The creation-time check that a `type: task` action's target task exists
   now runs for **every** `type: task` reference — a plain, same-workspace
   task name included — and regardless of whether the step has a `when`
   guard: an unknown workspace, a missing task, or a self-reference is a
   `400` at submit (`500` if the owner workspace is configured but currently
   unavailable). Before this release, only a cross-workspace (qualified)
   reference got this treatment; a plain reference to a missing task used to
   fail the step later, at dispatch. The separate check for a **bad literal
   connection value** on the task's schema is exempt for a `when`-guarded
   step (the step may never run) — it is a `400` at submit only when the
   step has no `when`; otherwise it falls back to failing the step at
   dispatch if reached, same as a templated value. A hook action wrapping
   a cross-workspace task is not created at all — see below.
6. The job detail step DTO carries `child_jobs` for `type: task` steps.
7. A cross-workspace `type: task` step's own job-step `input` (visible on
   the **parent** job) no longer shows the action owner's resolved default
   values when the owner is a different workspace from the caller — only
   what the caller itself supplied. The child job's input is unaffected and
   still carries the full merged input.

Action input defaults are rendered exactly once. A default whose render
produces `{{ … }}` text (for example `default: "{{ secret.X }}"` where
secret `X`'s value contains `{{`) keeps that text literally; it is never
evaluated a second time.

## Pod manifest overrides

Actions that run as Kubernetes pods (`type: pod` or `type: script` + `runner: pod`) support a `manifest` field for injecting arbitrary pod configuration. The value is deep-merged into the generated pod manifest.

### Merge rules

- **Objects**: recursively merged
- **Arrays of objects with `name` field** (e.g., `containers`, `env`): matched by `name` and deep-merged per element; unmatched entries are appended
- **Other arrays and scalars**: replaced entirely

### Examples

**Service account and annotations:**

```yaml
actions:
  deploy:
    type: pod
    image: company/deploy:latest
    cmd: "deploy --env production"
    manifest:
      metadata:
        annotations:
          iam.amazonaws.com/role: my-role
      spec:
        serviceAccountName: my-sa
```

**Resource limits (target the `step` container by name):**

```yaml
actions:
  heavy-build:
    type: pod
    image: node:20
    cmd: "npm run build"
    manifest:
      spec:
        containers:
          - name: step
            resources:
              requests:
                memory: "256Mi"
                cpu: "500m"
              limits:
                memory: "512Mi"
```

**Node selector and tolerations:**

```yaml
actions:
  train-model:
    type: pod
    image: pytorch/pytorch:2.1.0
    tags: ["gpu"]
    cmd: "python train.py"
    manifest:
      spec:
        nodeSelector:
          gpu: "true"
        tolerations:
          - key: "gpu"
            operator: "Exists"
            effect: "NoSchedule"
```

**Adding a sidecar container:**

```yaml
actions:
  test-with-db:
    type: pod
    image: node:20
    cmd: "npm test"
    manifest:
      spec:
        containers:
          - name: postgres-sidecar
            image: postgres:16
            env:
              - name: POSTGRES_PASSWORD
                value: test
```

The `manifest` field is rejected on `type: docker`, `type: task`, and `type: script` with `runner: local` or `runner: docker`.

## Structured output

Actions can emit structured output by printing a line with the `OUTPUT:` prefix followed by JSON:

```bash
#!/bin/bash
echo "Doing work..."
echo "OUTPUT: {\"status\": \"deployed\", \"version\": \"1.2.3\"}"
```

Only the **last** `OUTPUT: {json}` line is captured. The JSON is parsed and made available to downstream steps via [templating](https://fremvaerk.github.io/stroem/guides/templating/).

## Approval gates

Actions of `type: approval` pause job execution and request human approval before continuing. The approver can approve or reject the step, optionally providing a rejection reason.

**NOTE:** Approval gates are dispatched server-side — workers never see them and no worker tags are required.
### Approval lifecycle

1. **Pending**: Step is ready to execute but waiting for approval.
2. **Suspended**: Step is paused and awaiting human decision. Triggers `on_suspended` hooks (see [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/#on_suspended-hooks)).
3. **Completed** or **Failed**: Approver has made a decision; the step transitions to the final state and the job continues.

### Basic approval

```yaml
actions:
  deploy-approval:
    type: approval
    message: "Approve deployment to production?"

tasks:
  deploy:
    flow:
      approve:
        action: deploy-approval
      execute:
        action: deploy-script
        depends_on: [approve]
```

### Dynamic message with templating

The `message` field is a Tera template with access to job input and completed step outputs. Use it to show approvers relevant context:

```yaml
actions:
  deploy-approval:
    type: approval
    message: |
      Approve deployment of **{{ input.version }}** to **{{ input.environment }}**?

      Current version: {{ input.current_version }}
      Deployer: {{ input.deployer_email }}

tasks:
  deploy:
    input:
      version: { type: string, required: true }
      environment: { type: string, required: true }
      current_version: { type: string, required: true }
      deployer_email: { type: string, required: true }
    flow:
      approve:
        action: deploy-approval
        input:
          version: "{{ input.version }}"
          environment: "{{ input.environment }}"
          current_version: "{{ input.current_version }}"
          deployer_email: "{{ input.deployer_email }}"
      execute:
        action: deploy-script
        depends_on: [approve]
        input:
          version: "{{ input.version }}"
          environment: "{{ input.environment }}"
```

### Approve or reject via API

After a step enters `suspended` state, approvers use the API to make a decision:

**Approve:**
```bash
curl -X POST "http://localhost:8080/api/jobs/{job_id}/steps/{step_name}/approve" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"approved": true}'
```

**Reject:**
```bash
curl -X POST "http://localhost:8080/api/jobs/{job_id}/steps/{step_name}/approve" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"approved": false, "rejection_reason": "Version mismatch — need to use 2.0 instead"}'
```

The `rejection_reason` is optional but recommended for audit trails. When rejected, the step fails. Steps after a rejected approval run only if their own `depends_on` edge to the approval step accepts `failed` (e.g. `accept: [completed, failed]` or `accept: any`) — otherwise they're skipped `unreachable`. The approval step's own `continue_on_failure` is a separate decision: it's what decides whether the rejection fails the job, not whether anything downstream runs.

### Timeout behavior

Suspended steps are subject to a timeout. If not approved or rejected within the timeout period, the recovery sweep automatically fails the step:

```yaml
actions:
  deploy-approval:
    type: approval
    message: "Approve deployment?"
    timeout: 24h  # Max 24 hours — default is no timeout
```

Timed-out steps fail with a message like `"Approval step timed out after 24h"`. Steps after it run only if their own `depends_on` edge accepts `failed` — otherwise they're skipped `unreachable`. The approval step's own `continue_on_failure` only decides whether that timeout fails the job.

### Using on_suspended hooks

When a step enters `suspended` state, `on_suspended` hooks fire to notify approvers. See [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/#on_suspended-hooks) for full documentation and examples.

---

## Input & Output

## Input parameters

Both actions and tasks can declare input parameters.

```yaml
input:
  user_name:
    type: string          # single-line text
    name: User name       # optional, human-readable label in UI
    description: The user's full name  # optional, shown as helper text
    required: true         # fails if not provided
  env:
    type: string
    name: Environment
    description: Target deployment environment
    default: "staging"     # used when not provided
  query:
    type: text            # multiline text (textarea in UI)
    description: SQL query to execute
```

The optional `name` field provides a human-readable label for the input field in the web UI. When not set, the YAML key (e.g. `user_name`) is used as the label.

The optional `description` field is displayed in the web UI as helper text below the input field. The placeholder inside an empty input shows the field key (or "Select …" for option fields), never the description.

### Supported types

| Type       | Description                                                    |
|------------|----------------------------------------------------------------|
| `string`   | Single-line text. Renders as an input field in the UI          |
| `text`     | Multiline text. Renders as a textarea in the UI                |
| `integer`  | Whole number                                                   |
| `number`   | Numeric value (integer or decimal)                             |
| `boolean`  | True/false. Renders as a checkbox in the UI. Also accepts `bool` as an alias |
| `date`     | Date value (`YYYY-MM-DD`). Renders as a date field in the UI   |
| `datetime` | Date and time (`YYYY-MM-DDTHH:MM`). Renders as a date field plus a time field in the UI |
| `json`     | Any JSON value (object, array, string, number, boolean, null). Renders as a JSON editor in the UI |
| `file`     | An uploaded file. The step gets a read-only path to it. Renders as a file picker in the UI. See [File Inputs](https://fremvaerk.github.io/stroem/guides/file-inputs/) |

A `file` field cannot have `default`, `options`, `allow_custom`, `multiple` or `secret`. In a template, `{{ input.<field> }}` renders to a path, not to the file content.

If the `type` is not one of the types above, it is treated as a [connection type](https://fremvaerk.github.io/stroem/guides/connections/) reference. The UI renders a searchable dropdown of matching connections in the workspace.

Both `string` and `text` are treated identically at runtime — the difference is only in how the UI renders the input field. Use `text` for values that benefit from multiline editing such as SQL queries, scripts, or markdown content.

Both `date` and `datetime` are treated as strings at runtime. The `date` type produces values like `2026-01-15`, while `datetime` produces values like `2026-01-15T14:30`.

In the UI, type the date as `YYYY-MM-DD` (for example `2026-01-15`; `2026-1-5` is accepted and tidied to `2026-01-05`), or pick it from the calendar button at the end of the field. The calendar has month and year dropdowns (100 years back to 20 years ahead) and a **Today** button. Text that isn't a real date, such as `2026-02-31`, is marked invalid and the form won't submit until it is fixed or cleared.

### Task-level input

Task input is provided when triggering the task via API, CLI, or trigger:

```bash
curl -X POST http://localhost:8080/api/workspaces/default/tasks/deploy-pipeline/execute \
  -H "Content-Type: application/json" \
  -d '{"input": {"env": "production"}}'
```

Or via CLI:

```bash
stroem trigger deploy-pipeline --input '{"env": "production"}'
```

### Default values

Fields with a `default` are automatically filled in when not provided by the caller. This works across all job creation paths (API, CLI, triggers, webhooks, hooks, and task actions).

Default values can use [Tera templates](https://fremvaerk.github.io/stroem/guides/templating/) with access to `secret.*` (workspace secrets):

```yaml
tasks:
  deploy:
    input:
      env:
        type: string
        default: "staging"
      api_key:
        type: string
        default: "{{ secret.DEPLOY_API_KEY }}"
    flow:
      run:
        action: deploy-app
        input:
          env: "{{ input.env }}"
          api_key: "{{ input.api_key }}"
```

When triggered without specifying `env` or `api_key`, the defaults are applied automatically. Non-string defaults (numbers, booleans) pass through unchanged.

Fields marked `required: true` without a default will produce an error if not provided.

#### Empty fields in the UI

A field left empty in the **Run task** form is treated the same way as an API call that omits the key. If the field has no `default`, the key is not sent at all, so `{{ input.field | default(value='x') }}` applies the fallback and a plain `{{ input.field }}` fails to render, exactly as it would for an API, CLI or MCP caller that left the field out. If the field *has* a `default`, clearing it sends an empty string, which is an explicit override.

Tera's `default` filter fires for an *absent* or `null` variable, never for an empty string, so a template that must tolerate both should test the value: `{% if input.field %}{{ input.field }}{% else %}x{% endif %}`.

### JSON inputs

A `json` field holds any JSON value and keeps it structured through templates.

```yaml
actions:
  deploy:
    type: script
    script: echo "{{ input.cfg.region }} x{{ input.replicas + 1 }}"
    input:
      cfg: { type: json }
      replicas: { type: json }
tasks:
  release:
    input:
      targets: { type: json, default: [eu, us] }
    flow:
      plan: { action: make-plan }
      go:
        action: deploy
        depends_on: [plan]
        input:
          cfg: "{{ plan.output.cfg }}"            # an object
          replicas: "{{ plan.output.hosts | length }}"   # a number
```

Rules for every string inside a `json` value (also inside its objects and arrays):

- **Literal text** (no `{{`, `{%` or `{#`) is used as is.
- **Exactly one `{{ expression }}`** takes the expression's value — object, array, number, boolean, string or `null`. A missing field (`{{ obj.missing }}`) or undefined variable (`{{ typo }}`) is an error, as everywhere; use `{{ obj.missing | default(value=none) }}` for an optional value (gives `null`).
- **Anything else** (`"id {{ x }}"`, two expressions, `{% if %}` blocks) is an error. Build text inside one expression instead: `{{ 'id ' ~ x }}`.
- `| json_encode()` gives JSON *text* — the field then holds a string. Leave it out to pass the value; `stroem validate` warns about it.
- **Known limitation:** a `}}` or `{{` inside a string literal of the expression (e.g. `{{ x | default(value="}}") }}`) makes the string count as more than one expression. Use a variable instead.

`json_encode` belongs in a script body, not in the field. To pass a whole json input to a script as JSON text, write `{{ input.cfg | json_encode() }}` there; a bare `{{ input.cfg }}` in a script renders Tera's own format, which is not JSON.

Values sent through the API, a webhook's `body`, the CLI `--input`, MCP and agent tools are used exactly as given. A `json` field cannot be `secret`, have `options`/`allow_custom`/`multiple`, or be used in an approval form.

In the **Run task** form a `json` field is a JSON editor with syntax highlighting (job and step input/output on the job page are highlighted too). A default that contains templates is shown read-only ("evaluated when the job runs"); *Override* opens an empty editor. An empty editor sends nothing (the default applies) — type `""` or `null` to send those values. On a re-run the field starts in *replay*: the previous run's value is reused exactly; *Edit* opens it in the editor. A value typed in the form is parsed by the browser, so integers beyond 2^53 lose precision there — use the API or replay for those.

### Secret inputs

Mark an input as `secret: true` to indicate it contains sensitive data:

```yaml
tasks:
  deploy:
    input:
      api_key:
        type: string
        secret: true
        default: "{{ secret.DEPLOY_API_KEY }}"
```

When `secret: true` is set and the field has a default value:

- The UI renders a password field with a masked placeholder (`********`)
- If the user submits without changing the value, the field is omitted from the input payload
- The server fills in the default value and Tera renders the secret reference at execution time

This prevents the raw template string (e.g. `{{ secret.DEPLOY_API_KEY }}`) from appearing in the UI while ensuring the real secret value is used at runtime.

### Dropdown options

Add `options` to any input field to render it as a dropdown in the UI:

```yaml
tasks:
  deploy:
    input:
      env:
        type: string
        options: [staging, production, dev]
        default: staging
      region:
        type: string
        name: AWS Region
        options:
          - us-east-1
          - eu-west-1
          - ap-southeast-1
        allow_custom: true
```

When `options` is set, the UI renders a select dropdown instead of a text input. The value submitted is always a string from the list.

Set `allow_custom: true` to let users type a custom value in addition to the predefined options. The UI renders a text input with autocomplete suggestions.

### Multi-select

Set `multiple: true` (alongside `options`) to let users pick more than one value. The UI renders a searchable popover with a checkbox per option, and the submitted value is a JSON array.

```yaml
tasks:
  deploy:
    input:
      environments:
        type: string
        options: [dev, staging, prod, sandbox]
        multiple: true
        default: [dev, staging]
        required: true
        description: Target environments
```

Notes:

- `default`, if present, must be a JSON array. Each item must appear in `options` unless `allow_custom: true`.
- `allow_custom: true` works the same way as for single-select — users can type values not in the list and add them to the selection.
- `multiple: true` is not supported on connection-type fields, and cannot be combined with `secret: true`.
- Inside templates the value is a list — use Tera's array filters, e.g. `{{ input.environments | join(sep=', ') }}` or iterate with `{% for env in input.environments %}`.

### Display order

By default, input fields appear in YAML key order (which may vary since YAML maps are unordered). Use `order` to control the display order in the UI:

```yaml
tasks:
  report:
    input:
      end_date:
        type: date
        name: End date
        order: 2
      start_date:
        type: date
        name: Start date
        order: 1
      format:
        type: string
        default: csv
```

Fields with lower `order` values appear first. Fields without `order` appear after all ordered fields, in their original key order.

### Action-level input

Action input is provided by the step definition in the task flow. Values support [Tera templates](https://fremvaerk.github.io/stroem/guides/templating/):

```yaml
tasks:
  deploy:
    input:
      env: { type: string, default: "staging" }
    flow:
      deploy-step:
        action: deploy
        input:
          env: "{{ input.env }}"
```

## Structured output

Actions can emit structured output by printing a line with the `OUTPUT:` prefix followed by JSON:

```bash
#!/bin/bash
echo "Doing work..."
echo "OUTPUT: {\"status\": \"deployed\", \"version\": \"1.2.3\"}"
```

Only the **last** `OUTPUT: {json}` line is captured. The JSON is parsed and made available to downstream steps via templating:

```yaml
flow:
  build:
    action: build-app
  deploy:
    action: deploy
    depends_on: [build]
    input:
      version: "{{ build.output.version }}"
```

## Environment variables

Actions can declare environment variables. Values support templating:

```yaml
actions:
  deploy:
    type: script
    script: actions/deploy.sh
    env:
      DEPLOY_ENV: "{{ input.env }}"
      API_KEY: "{{ secret.api_key }}"
    input:
      env: { type: string }
```

## Organizing tasks with folders

Tasks can be organized into folders using the optional `folder` property. The UI displays tasks in a collapsible folder tree when folders are present.

### Basic folder

```yaml
tasks:
  deploy-staging:
    folder: deploy
    flow:
      run:
        action: deploy-app
```

### Nested folders

Use `/` to create nested folder hierarchies:

```yaml
tasks:
  deploy-staging:
    folder: deploy/staging
    flow:
      run:
        action: deploy-app

  deploy-production:
    folder: deploy/production
    flow:
      run:
        action: deploy-app

  run-etl:
    folder: data/pipelines
    flow:
      extract:
        action: extract-data
```

This creates a tree structure in the UI:

```
deploy/
  staging/
    deploy-staging
  production/
    deploy-production
data/
  pipelines/
    run-etl
```

Tasks without a `folder` property appear at the root level. When no tasks have folders, the UI shows a flat table.

---

## Templating

Strøm uses [Tera 2](https://keats.github.io/tera/) for templating. Templates are rendered on the server (when a worker claims a step, and for task actions, `when`, `for_each`, hooks and approval messages) and by the `stroem` CLI — never on the worker itself. Coming from Tera 1? See the [upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-tera-2/).

## Available context

Every template field a step can carry sees the same variables. There is one
rule, not one per field.

| Variable | Value | Notes |
|---|---|---|
| `input.*` | job input — or, in action bodies (`script`, `cmd`, `env`, `args`, `manifest`, `image`), the step's resolved input | approval `message:` sees the step's resolved input when the step has an input mapping, else job input |
| `secret.*` | workspace secrets (the action's **owner** workspace in action bodies; the job's workspace elsewhere) | always present, `{}` when none |
| `state.*` / `global_state.*` | the latest task / global state snapshot's `state.json` | `null` until a parsed snapshot exists — `{{ not state }}` is true whenever no parsed `state.json` is available (no snapshot yet, a snapshot without a sidecar, or one whose sidecar did not parse), and `{{ state.x \| default(value=0) }}` falls back to the default |
| `job.revision` | workspace revision pinned at creation — for a [pinned job](https://fremvaerk.github.io/stroem/guides/git-refs/), the commit its ref resolved to | always present, `null` for pre-migration jobs |
| `job.ref` | the [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) a pinned job runs at, as written (`release/2.3`, `v4.1.0`, a commit SHA) | always present, `null` (renders as an empty string) for every job that does not run on a ref |
| `<step>.output` | a finished step's output (`null` when a completed step produced none; `null` for skipped, failed and suspended steps — either renders as an empty string) | hyphens in step names become underscores |
| `<step>.error` | a failed step's error message | |
| `each.item` / `each.index` / `each.total` | loop variables inside a `for_each` instance | not available in `when:` — the condition runs before the loop expands |

**Reserved names.** A flow step named `input`, `secret`, `state`,
`global_state` or `job` shadows that variable; a step named `each` is shadowed
by the loop variable. When a worker claims the step, the server writes a
`[render] step '…' shadows template variable '…'` line to the job log; for
`when:`, `for_each:`, `type: task` inputs and approval messages the collision
is recorded in the server log only. Avoid these names.

A step named after a Tera 2 keyword — `none`, `null`, `self`, `loop`, `break`,
`continue`, `true`, `false`, `and`, `or`, `not`, `is`, `in`, `if`, `else` —
cannot be referenced from a template at all. The server logs
`[render] step '…' is a Tera keyword and cannot be referenced in templates`.
Rename the step.

## Basic usage

```yaml
actions:
  greet:
    type: script
    script: "echo Hello {{ input.name }}"
    input:
      name: { type: string, required: true }
```

## Passing data between steps

When a step emits structured output (via `OUTPUT: {json}`), downstream steps can reference it in templates.

```yaml
actions:
  greet:
    type: script
    script: "echo Hello {{ input.name }} && echo 'OUTPUT: {\"greeting\": \"Hello {{ input.name }}\"}'"
    input:
      name: { type: string, required: true }

  shout:
    type: script
    script: "echo {{ input.message }} | tr '[:lower:]' '[:upper:]'"
    input:
      message: { type: string, required: true }

tasks:
  hello-world:
    mode: distributed
    input:
      name: { type: string, default: "World" }
    flow:
      say-hello:
        action: greet
        input:
          name: "{{ input.name }}"
      shout-it:
        action: shout
        depends_on: [say-hello]
        input:
          # say-hello -> say_hello (hyphens become underscores)
          message: "{{ say_hello.output.greeting }}"
```

## Job metadata

Every job pins the workspace revision (git commit SHA for git workspaces, content hash for folder workspaces) at creation time. It is available as `{{ job.revision }}` in step inputs, `when` conditions, action bodies (`script`, `cmd`, `env`, `args`, `image`, `manifest`), agent prompts, and approval messages:

```yaml
actions:
  deploy:
    type: docker
    # Deploy the image built from the exact commit this job runs at
    image: "my-registry/app:{{ job.revision }}"

  report:
    type: script
    script: "echo Deployed revision {{ job.revision }}"
```

The value is identical for every step of a job (sub-jobs and hook jobs inherit the parent's revision — except a child that runs another workspace's task, or its own [`ref:`](https://fremvaerk.github.io/stroem/guides/git-refs/)). For jobs created before revision tracking existed it renders as an empty string. In hooks, the same value is available as `hook.revision` — see [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/).

`{{ job.ref }}` is the ref a [pinned job](https://fremvaerk.github.io/stroem/guides/git-refs/) runs at —
`release/2.3`, `v4.1.0`, or a commit SHA as written — and an empty string
for every other job; `{{ job.revision }}` of a pinned job is the resolved
commit. Hooks get the same value as `hook.ref`.

```yaml
actions:
  report:
    type: script
    script: "echo Running {% if job.ref %}ref {{ job.ref }}{% else %}the default branch{% endif %} at {{ job.revision }}"
```

**NOTE:** A flow step literally named `job` shadows the job metadata: `{{ job.output.* }}` keeps referring to that step's output, and `{{ job.revision }}` is unavailable in that task. Avoid naming a step `job` if you want the metadata.
## Templated refs

A `ref:` on a flow step, a `type: task` action or a webhook trigger can be a
template ([Git refs → Templated refs](https://fremvaerk.github.io/stroem/guides/git-refs/#templated-refs)). Its
context is narrower than a step input's:

| `ref:` on | Context |
|---|---|
| a flow step | `input`, `state`, `global_state`, `job`, earlier steps, `each` |
| a `type: task` action | `input` (the step's input rendered as strings), `job` |
| a webhook trigger | `input` (the webhook job input) |

None of them has `secret`, and `vals` is an unknown filter there. A value
that renders empty, or only whitespace, means no ref: the step, the task or
the webhook job runs as if the `ref:` key were absent.

## Step name rules

**CAUTION:** Step names with hyphens (e.g., `say-hello`) are sanitized to underscores (`say_hello`) in the template context because Tera interprets hyphens as subtraction.
- Step names in YAML can use hyphens: `say-hello`
- In template references, use underscores: `{{ say_hello.output.* }}`

## Tera features

Tera supports filters, conditionals, and more:

```yaml
# Filters
script: "echo {{ name | upper }}"
script: "echo {{ name | default(value='World') }}"

# Conditionals
script: "{% if enabled %}echo Active{% else %}echo Inactive{% endif %}"
```

See the [Tera 2 documentation](https://keats.github.io/tera/) for the full feature set.

### Undefined values and `default`

- Only the **last** path segment may be undefined: `{{ a.b | default(value=1) }}`
  works when `b` is missing, `{{ a.b.c | default(value=1) }}` is an error when
  `b` is missing. Use optional chaining: `{{ a.b?.c | default(value=1) }}`.
- `default` replaces both undefined **and `null`** values, so
  `{{ step.output.x | default(value='n/a') }}` works over a skipped, failed or
  suspended step (whose `output` is `null`). Strøm keeps this Tera 1 behaviour on
  top of Tera 2.
- `state` and `global_state` are `null` before the first snapshot, so
  `state.x | default(value=0)` is safe on a task's first run.
- `and` / `or` return one of their operands, not a boolean.
- Render arrays and objects with `| json_encode()` when you need JSON; a bare
  `{{ obj }}` renders `{"k": v}` and a string array renders `["a", "b"]`.

In a `json` input field, a string that is exactly one `{{ expression }}` keeps the expression's native value — see [JSON inputs](https://fremvaerk.github.io/stroem/guides/input-and-output/#json-inputs).

### Template errors

Template errors are **value-free**: they carry a category, the position
(`line L, column C`) and at most a type name or the failing filter's name —
never the rendered value, a variable name or the template line — so they are
safe in job logs, step errors and API responses. For the full Tera report run
the same workspace locally with `stroem run` or `stroem validate`. Limits: `stroem validate` compiles only `when`, `for_each` and agent prompts (plus secrets and connections at load), and `stroem run` only runs tasks made entirely of local `type: script` steps.

The Tera 1 filters `urlencode`, `urlencode_strict`, `slugify`,
`filesizeformat`, `striptags`, `addslashes` and `get_env`, plus the `spaceless` filter, the `is matching`
test and `get_random()`, are not available;
see the [upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-tera-2/#11-filters-that-were-not-restored).

## Input defaults with templates

Task input defaults support Tera templates with access to `secret.*`. Defaults are rendered at job creation time, before the job is persisted:

```yaml
tasks:
  deploy:
    input:
      api_key:
        type: string
        default: "{{ secret.DEPLOY_KEY }}"
```

See [Input & Output](https://fremvaerk.github.io/stroem/guides/input-and-output/) for full details on default values.

## Secret references in templates

The `| vals` filter resolves secret references at template render time. See [Secrets & Encryption](https://fremvaerk.github.io/stroem/guides/secrets/) for details.

```yaml
env:
  DB_PASSWORD: "{{ 'ref+awsssm:///prod/db/password' | vals }}"
```

## Conditional step execution

Steps support a `when` field for conditional execution. The condition is a Tera template that evaluates to true or false when the step's dependencies are met:

```yaml
tasks:
  conditional:
    input:
      run_checks: { type: boolean, default: false }
    flow:
      check:
        action: validate
        when: "{{ input.run_checks }}"

      process:
        action: process-data
        depends_on:
          - step: check
            accept: [completed, skipped]
        # Runs whether check ran or was skipped — the accept set that lets
        # it through lives on `process`'s own edge, the dependent, not on
        # `check`.
```

See [Conditional Flow Steps](https://fremvaerk.github.io/stroem/guides/conditionals/) for the full feature documentation.

---

## Conditional Flow Steps

Steps can be conditionally executed based on dynamic expressions evaluated at runtime, and a step's `depends_on` entries control exactly which of its dependencies' outcomes let it run.

## Overview

Two independent mechanisms work together:

- **`when`**: a Tera template expression on a step itself, evaluated once its dependency tree is satisfied. Truthy → runs; falsy → skipped (reason `condition`); empty `for_each` → skipped (reason `empty`); template error → the step **fails**.
- **`depends_on`**: a tree of dependencies this step waits on. Each entry names which of its dependency's terminal **outcomes** — `completed`, `failed`, `cancelled`, `skipped`, or `omitted` — satisfy that edge. A step runs only when its whole `depends_on` tree is satisfied; `when` is then evaluated on top of that, never instead of it.

A bare step name (`depends_on: [a]`) is sugar for "must complete" (`accept: [completed]`) — today's default, unchanged for any workflow that doesn't need anything else. To tolerate something other than a clean completion, write the dependency as an object with an explicit `accept` set:

```yaml
depends_on:
  - a                                  # sugar for {step: a, accept: [completed]}
  - step: b
    accept: [completed, failed]        # tolerates b failing, not being skipped
  - step: c
    accept: any                   # ordering only — don't care how c ends
```

**Readiness waits for everything named, every time.** Whether a dependency is required or tolerated, the step is only decided once *every* step named anywhere in its `depends_on` tree has gone terminal — there's no fail-fast on an already-unsatisfiable tree, and no shortcut for an `any` group already satisfied by its first child. See [Validation](#validation) and the [YAML reference](https://fremvaerk.github.io/stroem/reference/workflow-yaml/#dependencies) for the full grammar, including `all`/`any` grouping.

**Validation**: `when` syntax and `depends_on` shape are validated at YAML parse time — a typo'd key (`accpet` instead of `accept`) or a mapping with two discriminating keys at once (`step` and `any` together) is a hard error, not a silent default.

:::note[Upgrading from 0.17 or earlier]
If your workflows still use `continue_on_failure` to decide whether a *dependent* runs, or `continue_when_skipped` at all, read the [0.18 upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-0-18-dependency-conditions/) first — the model below replaces both.
:::

## YAML Syntax

```yaml
tasks:
  conditional-task:
    input:
      run_checks: { type: boolean, default: false }
    flow:
      setup:
        action: init-workspace

      check-data:
        action: validate-input
        depends_on: [setup]
        when: "{{ input.run_checks }}"

      process:
        action: process-data
        depends_on:
          - setup
          - step: check-data
            accept: [completed, skipped]
            # Lets `process` through whether check-data ran or was
            # skipped by its own `when` — but not if it failed.
```

## Truthiness Rules

Tera templates render to strings, and Strøm decides truthiness on the **rendered text** (after trimming). The following are **falsy** and cause the step to be skipped:

| Value | Skipped? |
|-------|----------|
| Empty string `""` | Yes |
| `"false"` (case-insensitive: `"False"`, `"FALSE"`, etc.) | Yes |
| `"0"` | Yes |
| `"null"` (case-insensitive: `"Null"`, `"NULL"`, etc.) | Yes |
| `"none"` (case-insensitive: `"None"`, `"NONE"`, etc.) | Yes |
| `"[]"` and `"{}"` (an empty array or object) | Yes |
| Any number equal to zero: `"0"`, `"0.0"`, `"-0.0"` | Yes |
| `"true"`, `"1"`, any other string | No |
| Template error (e.g., undefined variable) | Fails the step |

Worked cases (with `z = 0.0`, `e = []`, `m = {}`, `s = "false"`, `n = null`, `one = 1`):

| `when:` | Result |
|---|---|
| `{{ z }}`, `{{ e }}`, `{{ m }}`, `{{ s }}`, `{{ n }}` | skipped |
| `{{ e and one }}`, `{{ one and e }}` | skipped (`and` returns the empty operand) |
| `{{ e or one }}` | runs (`or` returns `1`) |
| `{{ one }}`, `{{ [0] }}`, `0.5`, `x` | runs (`[0]` is a non-empty array) |
| empty string, `0`, `-0.0`, `None`, `NULL` | skipped |

**NOTE:** A condition that renders an empty array (`when: "{{ scan.output.items }}"`) is
now **false**; under Tera 1 it rendered `[]` and was truthy.
**CAUTION:** Empty strings evaluate to falsy. Use `{{ input.value \| default(value='') }}` carefully — it will skip the step if undefined.
## Available Template Variables

Inside a `when` expression, you can reference:

| Variable | Description |
|----------|--------------|
| `input.*` | Job-level input from the API call or trigger |
| `<step_name>.output.*` | Output from a completed upstream step |
| `secret.*` | Workspace secrets (after rendering) |

**Step name rules**: Step names with hyphens become underscores in templates. A step named `check-data` is referenced as `check_data.output.*`.

### Example: Referencing step outputs

```yaml
actions:
  check-status:
    type: script
    script: |
      status=$(curl -s https://api.example.com/status)
      echo "OUTPUT: {\"ok\": $(echo $status | jq '.healthy')}"
    output:
      ok: { type: boolean }

  do-something:
    type: script
    script: "echo Running..."

tasks:
  monitor:
    flow:
      check:
        action: check-status

      proceed:
        action: do-something
        depends_on: [check]
        # Use check_status's output (note: step name check becomes check, so check.output)
        when: "{{ check.output.ok }}"
```

## Mixed fan-in: required, tolerated, and ordering-only in one tree

Three dependencies on one step rarely all mean the same thing. `accept` lets each edge say exactly what it needs:

```yaml
tasks:
  publish-task:
    flow:
      source:
        action: fetch-source

      enrichment:
        action: enrich-data
        depends_on: [source]

      audit:
        action: audit-log
        depends_on: [source]

      publish:
        action: publish
        depends_on:
          - source                                 # must complete
          - step: enrichment
            accept: [completed, failed]              # tolerates a failed enrichment, not a skip
          - step: audit
            accept: any                          # pure ordering: wait for it, don't care how it ends
```

`enrichment` and `audit` each carry their own `continue_on_failure: true` if their own failure shouldn't fail the *job* — that's a separate decision (see [Accounting vs. readiness](#accounting-vs-readiness) below) from whether `publish` *accepts* their failure. This is the shape 0.17's flags had no answer for: "tolerate a dependency's failure, but still block on it being skipped" needs `[completed, failed]` — `skipped` and `omitted` just aren't in the set.

### `any`/`all` grouping

For "at least one of these must succeed," wrap dependencies in `any` (or `all` for an explicit, nestable AND):

```yaml
tasks:
  ranked-task:
    flow:
      audit:
        action: audit-log

      mirror-a:
        action: sync-to-mirror-a
        continue_on_failure: true   # its own failure doesn't fail the job

      mirror-b:
        action: sync-to-mirror-b

      ranked:
        action: rank-mirrors
        depends_on:
          - step: audit
            accept: any
          - any: [mirror-a, mirror-b]   # at least one mirror must complete
```

`ranked` runs once `audit` is terminal (whatever its outcome) **and** at least one of `mirror-a`/`mirror-b` completed — even if `mirror-a` fails, as long as `mirror-b` completes. Per the readiness rule above, `ranked` still waits for *both* mirrors to finish before deciding, even though the `any` is already satisfied once one of them completes — no short-circuiting ahead of a slower sibling.

`all`/`any` nest arbitrarily: an `any` can contain an `all`, a `step` entry, or a bare name. The same step name can legitimately appear in two different branches with different `accept` sets (e.g. "`A` and `B` both completed, OR `A` failed and `C` completed" needs `A` once per branch) — that's not a mistake, and it's accepted. What's rejected is the same name appearing twice as a **direct sibling** of the same `all`/`any` node (see [Validation](#validation)).

## Accounting vs. readiness

These are two independent questions, and `accept` only answers one of them:

- **Readiness** — does a dependent get to run? Answered entirely by `depends_on`'s `accept` sets, evaluated per edge.
- **Accounting** — does a step's own failure fail the *job*? Answered entirely by `continue_on_failure` on that step itself, self-scoped, with no propagation to anyone downstream.

```yaml
flow:
  last-step:
    action: do-something
    continue_on_failure: true   # its own failure doesn't fail the job

  cleanup:
    action: remove-temp-files
    depends_on:
      - step: last-step
        accept: any         # runs whether last-step completed, failed, or was cancelled
```

`continue_on_failure` on `last-step` means `last-step` failing doesn't fail the job — full stop. It has no effect on whether `cleanup` runs; that's `cleanup`'s own `accept: any`. The example above, as written (no `continue_on_failure` on `last-step`), already does the more direct thing: `cleanup` runs after a failure **and** the job still ends `failed` (so `on_error` fires too) — `accept` and `continue_on_failure` are fully decoupled, so a cleanup step doesn't need to leave the flow just to keep the job's outcome honest. Reach for an [`on_error` / `on_cancel` hook](https://fremvaerk.github.io/stroem/guides/hooks/) instead when what needs to run isn't really part of the task's own DAG — paging on-call, tearing down infrastructure the flow itself never touched — and you want it to fire exactly once on any terminal outcome without wiring an `accept` edge to every step that might fail.

A dependent's own `when` is still evaluated on its own terms once its `depends_on` tree is satisfied: the tree decides whether the dependent is even considered, not what its own condition renders to.

## Validation

- Every step name referenced anywhere in a `depends_on` tree — including nested inside `all`/`any` — must exist in the flow.
- A typo'd key in a `depends_on` entry (e.g. `accpet` instead of `accept`), or a mapping carrying two discriminating keys at once (e.g. both `step` and `any`), is a hard parse error.
- The same step name may appear more than once across *different* branches of a tree with different `accept` sets — that's a legitimate OR expression, not a mistake. It's rejected only as a **direct sibling** of the same `all`/`any` node (including the implicit top-level `all`): `depends_on: [a, a]`-style duplicates must be deduplicated.
- An empty `Outcomes` list, a duplicate entry within one `accept: [...]` list, or a nested `all: []`/`any: []`, is a validation error (an empty group would trivially always or never pass; a duplicate outcome is always redundant). The one exception: a step with **no** `depends_on` at all — field absent or explicitly `depends_on: []` — is unchanged from today and ready immediately.
- `continue_when_skipped` is retired. A task with a step that still sets it gets a named error pointing at the [0.18 upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-0-18-dependency-conditions/) instead of silent acceptance — from `stroem validate`, or at job-creation time (server) / before execution (`stroem run`). It is **not** a parse error: a server loading or reloading the workspace does not catch it on its own, only an actual job-creation attempt for the affected task does.

## Outcomes and skip reasons

Every dependency resolves, once terminal, to exactly one of five **outcomes** — this is the vocabulary `accept` sets are built from:

| Outcome | Maps to |
|---|---|
| `completed` | `status = completed` |
| `failed` | `status = failed` |
| `cancelled` | `status = cancelled` |
| `skipped` | `status = skipped`, reason `condition` or `empty` — the step's **own** choice (its `when` was false, or its `for_each` was empty) |
| `omitted` | `status = skipped`, reason `unreachable` (or a legacy `cascade` row from before 0.18, or an unrecognized/NULL reason) — blocked by **its own** dependency tree, not by its own choice |

The job detail page and the API both still show `skip_reason` on the step itself (`condition`, `empty`, or `unreachable`) — that's a diagnostic on the row, independent of what a *downstream* dependent's `accept` set does with it. An `omitted` step tells you "something in my own tree didn't let me through," not *what* — see the [0.18 upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-0-18-dependency-conditions/#structural-catch-audit) if you need to trace the actual origin several hops back.

## Root Step Conditions

A step with no `depends_on` is ready from the start. Its `when` condition is evaluated immediately at job creation time:

```yaml
tasks:
  root-conditional:
    input:
      skip_setup: { type: boolean, default: false }
    flow:
      setup:
        action: initialize
        when: "{{ not input.skip_setup }}"

      main:
        action: do-work
        depends_on:
          - step: setup
            accept: [completed, skipped]
        # Runs whether setup ran or was skipped by its own `when`.
```

## Error Handling

If a `when` expression **fails to render** (e.g., an undefined variable, an undefined operand of `and` / `or`, or a syntax error), the step fails (status = `failed`, not skipped). This prevents silent failures:

```yaml
tasks:
  error-condition:
    flow:
      check:
        action: validate

      proceed:
        action: next-step
        depends_on: [check]
        # If check.output.status doesn't exist, this step FAILS
        # (doesn't skip — you get an error to fix)
        when: "{{ check.output.status }}"
```

Comparing against a missing **field** is not an error: `{{ check.output.status == 'ok' }}` renders `false` when `status` is missing (`check.output` itself must still exist), so the step is skipped rather than failed. A missing *parent* (`a.b.c` with `b` missing) is still an error — use `a.b?.c`. Template errors never contain rendered values; run `stroem validate` or `stroem run` locally for Tera's full report. Limits: `stroem validate` compiles only `when`, `for_each` and agent prompts (plus secrets and connections at load), and `stroem run` only runs tasks made entirely of local `type: script` steps.

To handle undefined outputs gracefully, use Tera filters:

```yaml
when: "{{ check.output.status | default(value='') }}"
```

This will skip the step if `check.output.status` is undefined (renders to empty string).

## Common Patterns

### If/else branch

Two mutually exclusive branches that merge back together:

```yaml
tasks:
  if-else:
    input:
      mode: { type: string, enum: [fast, slow] }
    flow:
      fast-path:
        action: quick-process
        when: "{{ input.mode == 'fast' }}"

      slow-path:
        action: thorough-process
        when: "{{ input.mode == 'slow' }}"

      merge:
        action: finalize
        depends_on:
          - step: fast-path
            accept: [completed, skipped]
          - step: slow-path
            accept: [completed, skipped]
        # Exactly one branch runs, the other is skipped `condition`.
        # Both edges accept `skipped`, so merge sees every dependency
        # as satisfied either way.
```

### Multi-step branch

A condition gates a branch of multiple steps. Downstream steps only need `accept` where they actually need to tolerate a skip or failure — a plain `depends_on: [advanced-step-1]` still requires `completed`, and cascades to `omitted` if the branch root was skipped:

```yaml
tasks:
  multi-step-branch:
    input:
      enable_advanced: { type: boolean }
    flow:
      advanced-setup:
        action: advanced-init
        when: "{{ input.enable_advanced }}"

      advanced-step-1:
        action: advanced-processing-1
        depends_on: [advanced-setup]
        # Omitted if advanced-setup was skipped (default accept: [completed])

      advanced-step-2:
        action: advanced-processing-2
        depends_on: [advanced-step-1]
        # Also omitted if advanced-step-1 was omitted

      summary:
        action: generate-report
        depends_on:
          - step: advanced-step-2
            accept: [completed, omitted]
        # Runs even if the whole branch was never entered.
```

### Optional step (skip if not needed)

An optional step in a linear pipeline. Accept its `skipped` outcome on the edge so dependents run whether it runs or is skipped:

```yaml
tasks:
  input-conditional:
    input:
      skip_validation: { type: boolean, default: false }
      data: { type: string, required: true }
    flow:
      prepare:
        action: prepare-data
        input:
          data: "{{ input.data }}"

      validate:
        action: validate-data
        depends_on: [prepare]
        when: "{{ not input.skip_validation }}"

      process:
        action: use-data
        depends_on:
          - prepare
          - step: validate
            accept: [completed, skipped]
```

### Condition based on step output

Run a step only if a previous step's output meets a condition:

```yaml
actions:
  check-data:
    type: script
    script: |
      count=$(curl -s https://api.example.com/count)
      echo "OUTPUT: {\"count\": $count}"
    output:
      count: { type: integer }

  process-large-dataset:
    type: script
    script: "echo Processing large dataset..."

tasks:
  conditional-processing:
    flow:
      count-data:
        action: check-data

      process:
        action: process-large-dataset
        depends_on: [count-data]
        # Only run if count is >= 1000
        when: "{{ count_data.output.count >= 1000 }}"
```

## Example Workflow

A complete example combining multiple conditional patterns:

```yaml
actions:
  pre-check:
    type: script
    script: "echo 'Checking...' && echo 'OUTPUT: {\"ready\": true}'"
    output:
      ready: { type: boolean }

  fast-process:
    type: script
    script: "echo Fast processing"

  slow-process:
    type: script
    script: "echo Slow processing"

  cleanup:
    type: script
    script: "echo Cleanup"

tasks:
  full-example:
    input:
      use_fast: { type: boolean, default: true }
      skip_pre_check: { type: boolean, default: false }
    flow:
      verify:
        action: pre-check
        when: "{{ not input.skip_pre_check }}"

      fast:
        action: fast-process
        depends_on:
          - step: verify
            accept: [completed, skipped]
        when: "{{ input.use_fast }}"

      slow:
        action: slow-process
        depends_on:
          - step: verify
            accept: [completed, skipped]
        when: "{{ not input.use_fast }}"

      finish:
        action: cleanup
        depends_on:
          - step: fast
            accept: [completed, skipped]
          - step: slow
            accept: [completed, skipped]
```

This workflow:
1. Optionally runs the pre-check based on input. Both `fast` and `slow` accept `verify` being `skipped`, so a skip doesn't block either branch from evaluating its own `when`.
2. When `verify` runs (or is skipped), branches into fast or slow path based on input.
3. Converges at `finish` — exactly one of `fast`/`slow` runs, the other is skipped `condition`; `finish` accepts `skipped` on both edges, so it sees every dependency as satisfied.

**When do you still need a non-default `accept`?** Any time a dependent should proceed past something other than a clean `completed` — a dependency that failed, was cancelled, or was skipped (by choice or because its own tree blocked it). There's no automatic convergence: every edge of a merge needs its own `accept`, not just one of them.

---

## For-Each Loops

Steps can iterate over a list using `for_each`, running the same action for each item. This enables fan-out/fan-in patterns: expand a step into N parallel (or sequential) instances, then converge when all are done.

## Overview

- **Fan-out**: A `for_each` step is expanded into N instance steps at runtime, one per item in the list
- **Fan-in**: Downstream steps wait for all instances to complete. The placeholder step's output is an array of all instance outputs
- **Parallel by default**: All instances run concurrently. Use `sequential: true` for one-at-a-time execution
- **`each` variable**: Inside loop instances, `each.item` is the current array element and `each.index` is the zero-based index
- **Composable**: Combine with `when` conditions, `type: task` actions, and `continue_on_failure`

## YAML Syntax

### Parallel loop (default)

```yaml
actions:
  get-items:
    type: script
    script: echo 'OUTPUT: {"items": ["a","b","c"]}'
  process-item:
    type: script
    script: |
      echo "Processing {{ input.item }}"
      echo 'OUTPUT: {"result": "done-{{ input.item }}"}'

tasks:
  main:
    flow:
      fetch:
        action: get-items

      process:
        action: process-item
        depends_on: [fetch]
        for_each: "{{ fetch.output.items }}"
        input:
          item: "{{ each.item }}"
          index: "{{ each.index }}"

      aggregate:
        action: combine
        depends_on: [process]
        input:
          results: "{{ process.output }}"
```

When the job runs:
1. `fetch` completes and outputs `{ "items": ["a", "b", "c"] }`
2. `process` expands into `process[0]`, `process[1]`, `process[2]` — all run in parallel
3. `aggregate` waits for all 3 instances, then sees `process.output` as `[{"result":"done-a"}, {"result":"done-b"}, {"result":"done-c"}]`

### Rendering the array

`for_each` must render to a JSON array. Render arrays and objects with
`| json_encode()` — it always produces valid JSON, whereas a bare `{{ x }}`
renders Tera's display format (`{"k": v}` for objects, floats with `.0`):

```yaml
for_each: "{{ fetch.output.items | json_encode() }}"
```

### Literal array

```yaml
flow:
  deploy:
    action: deploy-to-region
    for_each: ["us-east-1", "eu-west-1", "ap-south-1"]
    input:
      region: "{{ each.item }}"
```

### Sequential loop

Use `sequential: true` to run instances one at a time, in order:

```yaml
flow:
  migrate:
    action: run-migration
    for_each: "{{ fetch.output.schemas }}"
    sequential: true
    input:
      schema: "{{ each.item }}"
```

In sequential mode:
- Instance `[0]` starts immediately; `[1]` starts after `[0]` completes, etc.
- If an instance fails (and `continue_on_failure` is not set) every remaining instance is skipped immediately and the loop step fails.
- Useful for ordered operations like database migrations

## The `each` Variable

Inside loop instances, two special variables are available:

| Variable | Type | Description |
|----------|------|-------------|
| `each.item` | any | The current array element for this iteration |
| `each.index` | number | Zero-based index (0, 1, 2, ...) |

These are available in `input` templates and action templates (env, script, cmd).

## Fan-In and Output Aggregation

When all loop instances complete, the placeholder step's output becomes an array of all instance outputs, ordered by index:

```yaml
flow:
  process:
    action: compute
    for_each: ["a", "b", "c"]

  summary:
    action: aggregate
    depends_on: [process]
    input:
      # Array of outputs: [output_0, output_1, output_2]
      all_results: "{{ process.output }}"
```

- Failed instances always contribute `null` to the output array — visible unconditionally, with or without `continue_on_failure` on the placeholder (a dependent's own `accept` decides whether it ever sees that output at all)
- Skipped instances also contribute `null`

## Combining with `when` Conditions

`when` is evaluated **before** `for_each`. If the condition is falsy, the step is skipped entirely (no expansion):

```yaml
flow:
  deploy:
    action: deploy-to-region
    for_each: "{{ fetch.output.regions }}"
    when: "{{ input.should_deploy }}"
    input:
      region: "{{ each.item }}"
```

## Sub-Job Fan-Out (`type: task` + `for_each`)

A `for_each` step can reference a `type: task` action — each iteration creates a separate child job:

```yaml
actions:
  deploy-env:
    type: task
    task: deploy-pipeline

tasks:
  main:
    flow:
      deploy-all:
        action: deploy-env
        for_each: ["staging", "prod"]
        input:
          environment: "{{ each.item }}"

  deploy-pipeline:
    input:
      environment: { type: string }
    flow:
      setup:
        action: init-env
      deploy:
        action: run-deploy
        depends_on: [setup]
```

Each instance creates a full child job with its own steps, logs, and lifecycle. The called task can itself use `for_each` — enabling nested iteration through separate job scopes.

## Error Handling

| Scenario | Behavior |
|----------|----------|
| Empty array | Step is skipped (not failed) |
| Non-array result | Step fails with error: `for_each must render a JSON array; the rendered text (N bytes) is not valid JSON …` or `… got a JSON object` (the error describes the shape only, never the rendered text) |
| Instance failure (parallel) | All instances run regardless (nothing to sequence); placeholder rolls up `failed` once all are done — regardless of `continue_on_failure` |
| Instance failure (sequential) | Without `continue_on_failure` on the placeholder, remaining pending instances are skipped immediately. *With* it, the loop keeps promoting instances past the failure. Either way, the placeholder rolls up `failed` once done — the flag no longer changes the rollup itself |
| Template error in `for_each` expression | Step fails with error |

`continue_on_failure` on the loop's own placeholder still decides whether a failed instance stops a *sequential* loop from promoting further instances (same as any `continue_on_failure` effect pre-0.18) — but it no longer changes the final rollup: any failed instance makes the placeholder roll up `failed`, always, cof or not. The flag's remaining effect on job status is also unchanged: it decides whether that failure fails the *job* — self-scoped, like any other step. A dependent that must still run despite a failed loop needs its own `accept: [completed, failed]` (or `accept: any`) on the edge to the placeholder; without it, the placeholder's `failed` status skips the dependent `unreachable` the same way any other untolerated failure would.

## Limits

- Maximum 10,000 items per `for_each` expression
- Step names must not contain `[` or `]` (reserved for instance naming)
- Nested `for_each` within the same task is not supported — use `type: task` for nesting

---

## Retry Mechanisms

Strøm provides powerful retry mechanisms for handling transient failures. You can retry individual steps with configurable backoff, or retry entire tasks by creating new jobs. This guide covers both strategies and shows when to use each.

## Overview

Strøm has two retry layers:

1. **Step-level retry**: Retries a failed step in-place, with optional backoff between attempts
2. **Task-level retry**: Retries the entire task by creating a new job when the task fails

Both can use **fixed** or **exponential** backoff, with optional **jitter** to prevent thundering herd.

## Step-Level Retry

Step-level retry retries a single step without re-running dependencies. The step is reset and allows a worker to claim and re-execute it. This is fast and focused, ideal for transient failures.

### Basic syntax

```yaml
tasks:
  main:
    flow:
      fetch:
        action: fetch-data
        retry:
          max_attempts: 3
          delay: "30s"
          backoff: fixed
```

`max_attempts` counts executions including the first; `max_attempts: 1` disables retry for that step.

**Behavior:**
- Attempt 1: Step runs and fails
- Wait 30s
- Attempt 2: Step resets and a worker claims it again
- If Attempt 2 fails, wait 30s again
- Attempt 3: Final attempt
- If all 3 fail, the step fails and downstream steps are skipped

### Backoff strategies

#### Fixed backoff (default)

Same delay between all retry attempts:

```yaml
retry:
  delay: "30s"
  backoff: fixed
```

Use when: Failures are truly transient and waiting a fixed time is sufficient (e.g., temporary network blip).

#### Exponential backoff

Delay doubles after each failure, up to a 64x cap:

```yaml
retry:
  delay: "5s"
  backoff: exponential
```

Delays: 5s, 10s, 20s, 40s, 80s (capped at 320s), 320s, ...

Use when: You expect increasing wait times help (e.g., service warming up, queue draining).

#### Jitter

Add 0–25% random variance to smooth out synchronized retries across many concurrent steps:

```yaml
retry:
  delay: "10s"
  backoff: exponential
  jitter: true
```

With jitter, each delay varies randomly:
- Nominal 10s → actual 7.5s–12.5s
- Nominal 20s → actual 15s–25s

Use when: Running many tasks in parallel (prevents thundering herd at retry windows).

### Example: Unreliable API calls

```yaml
actions:
  call-external-api:
    type: script
    script: |
      curl -f https://api.example.com/data || exit 1
      echo 'OUTPUT: {"status": "success"}'

tasks:
  fetch-and-process:
    flow:
      fetch:
        action: call-external-api
        retry:
          max_attempts: 5
          delay: "10s"
          backoff: exponential
          jitter: true

      process:
        action: process-data
        depends_on: [fetch]
        input:
          data: "{{ fetch.output }}"
```

The `fetch` step will retry up to 5 times with exponential backoff + jitter before failing the entire task.

### Step retry vs. action defaults

Actions can define default retry config, overridable per step:

```yaml
actions:
  unstable-task:
    type: script
    script: "flaky-operation"
    retry:
      max_attempts: 2

tasks:
  main:
    flow:
      step1:
        action: unstable-task
        # Uses action default: 2 max_attempts

      step2:
        action: unstable-task
        retry:
          max_attempts: 5  # Overrides action default
```

**Resolution:** Step-level retry takes precedence. If a step doesn't specify retry, it inherits from the action.

## Task-Level Retry

Task-level retry retries the **entire task** by creating a new job when the task fails. All steps restart from the beginning.

### Basic syntax

```yaml
tasks:
  deploy:
    retry:
      max_attempts: 2
      delay: "60s"
    flow:
      build:
        action: build-app
      test:
        action: run-tests
      deploy:
        action: deploy-app
```

**Behavior:**
- Original job fails (e.g., at `deploy` step)
- Wait 60s
- Server creates a **new job** with the same task and input, starting from `build`
- If new job fails, no further retries (max_attempts: 2 exhausted)
- `on_error` hooks fire only on final failure, not after each retry attempt

### Retry chain

Every retry job's `retry_of_job_id` points at the **first** job of the chain, and each failed attempt's `retry_job_id` points at the attempt that replaced it:

```
Job A (failed at deploy step)      retry_job_id → B
  └─ Job B (failed at test step)   retry_of_job_id → A, retry_job_id → C
      └─ Job C (success)           retry_of_job_id → A
```

In the UI, a retry job shows **retry of `<first job>`** under its title and its attempt number in the **Source** card; a failed attempt that was retried shows a **Retried** link to the next attempt. A job that you [restarted or re-ran](https://fremvaerk.github.io/stroem/guides/rerun-and-restart/#re-run) shows a **Later runs** card instead. If the first job has since been deleted by [retention](https://fremvaerk.github.io/stroem/operations/retention/), the pointer is cleared (`ON DELETE SET NULL`), and the next retry points at the attempt that just failed instead — not at the earliest attempt that still exists.

### When to use task retry

Use task-level retry when:
- Failures require starting from scratch (e.g., state cleanup)
- Dependencies need reloading (e.g., config refresh)
- Full job restart is acceptable (overhead is small)

Use step retry instead when:
- Failure is isolated to one step (don't restart everything)
- Each attempt is expensive (build steps, deployments)
- You want faster recovery

### Example: Deployment with state recovery

```yaml
tasks:
  deploy-service:
    retry:
      max_attempts: 2
      delay: "30s"
      backoff: exponential
    flow:
      check-health:
        action: check-service-health

      backup:
        action: backup-current-version
        depends_on: [check-health]

      deploy:
        action: deploy-new-version
        depends_on: [backup]

      verify:
        action: verify-deployment
        depends_on: [deploy]

    on_error:
      - action: notify-ops
        input:
          message: "Deploy failed after {{ hook.failed_steps | length }} step(s)"
```

If the deployment fails, the entire task retries — all steps run again with the latest version and state.

## Interactions with Other Features

### Retry + `continue_on_failure`

Step retry exhausts before anything else is decided — `continue_on_failure` and a dependent's `accept` are two separate questions about what happens once it has:

```yaml
flow:
  unstable:
    action: might-fail
    retry:
      max_attempts: 3
    continue_on_failure: true   # unstable's own failure doesn't fail the job

  next:
    action: next-step
    depends_on:
      - step: unstable
        accept: [completed, failed]   # runs even if all 3 retry attempts fail
```

If `unstable` exhausts retries and fails, `next` still runs because its own `depends_on` edge accepts `failed` — that's `next`'s decision, not `unstable`'s. `continue_on_failure` on `unstable` is unrelated to whether `next` runs; it only decides whether the exhausted failure fails the *job*.

### Retry + `for_each` loops

Loop instances inherit the step's retry config:

```yaml
flow:
  process-items:
    action: process
    for_each: ["item-a", "item-b", "item-c"]
    retry:
      max_attempts: 2
    input:
      item: "{{ each.item }}"

  # Each instance (process[0], process[1], process[2]) gets up to 2 attempts
```

If `process[1]` fails, it retries independently; `process[0]` and `process[2]` are unaffected.

### Retry + `when` conditions

Retry only applies if the step actually runs:

```yaml
flow:
  maybe-deploy:
    action: deploy
    when: "{{ input.deploy_now == true }}"
    retry:
      max_attempts: 3
    # If condition is false, step is skipped — no retry attempted
```

### Retry + timeouts

**Step timeout** applies per attempt. **Task timeout** applies to entire retry chain:

```yaml
tasks:
  main:
    timeout: 10m
    flow:
      slow-step:
        action: slow-operation
        timeout: 2m     # 2m per attempt
        retry:
          max_attempts: 3
          delay: "30s"
        # Possible timeline: 2m + 30s + 2m + 30s + 2m = 7m total
        # Well under 10m task timeout
```

If the task timeout is exceeded, all remaining retries are cancelled.

### Retry + hooks

Hooks fire **only on final failure** (after all retries exhausted), not after each retry attempt:

```yaml
tasks:
  main:
    flow:
      unstable:
        action: flaky-operation
        retry:
          max_attempts: 3

    on_error:
      - action: alert-ops
        # Fires only after all 3 attempts fail
```

On success (before exhausting retries), `on_success` hooks fire normally.

### Retry + task actions

Task-level retry does **not** retry child jobs created by `type: task` actions. Only the top-level task retries:

```yaml
tasks:
  parent:
    retry:
      max_attempts: 2
    flow:
      spawn-child:
        action: run-subtask
        # run-subtask is a type: task action

      next:
        action: next-step
        depends_on: [spawn-child]

  subtask:
    flow:
      work: { action: do-something }
```

If `subtask` fails, it does **not** trigger `parent`'s retry. To retry a child job, define retry on the individual steps within `subtask`.

## Limits and Constraints

| Constraint | Limit | Notes |
|-----------|-------|-------|
| `max_attempts` | 1–10 | 1 = no retries (attempt once), 10 = max 9 retries |
| `delay` | 0–3600s (1h) | Can be `0s` for immediate retry, or `1h` for long waits |
| Exponential backoff cap | 64x initial delay | Prevents unbounded growth |
| Step timeout per attempt | Max 24h | Applies to each individual attempt |
| Task timeout (entire retry chain) | Max 7 days | Includes all retry attempts and delays |

## Best Practices

### 1. Use step retry for isolated transient failures

```yaml
retry:
  max_attempts: 3
  delay: "5s"
  backoff: exponential
```

Examples: network timeouts, temporary rate limiting, brief service unavailability.

### 2. Use task retry for cascading or state-dependent failures

```yaml
retry:
  max_attempts: 2
  delay: "60s"
```

Examples: deployment rollback needed, config reload required, full health check.

### 3. Add jitter for high concurrency

```yaml
retry:
  delay: "10s"
  backoff: exponential
  jitter: true  # Spreads out retry waves
```

### 4. Avoid excessive retries

Too many attempts waste time:

```yaml
# Good
retry:
  max_attempts: 3
  delay: "10s"

# Avoid
retry:
  max_attempts: 10  # Likely a sign of deeper issue
  delay: "1s"
```

### 5. Combine with logging for observability

Include details in actions so you can debug:

```yaml
actions:
  call-api:
    type: script
    script: |
      echo "Attempt with timeout..." >&2
      curl --max-time 5 https://api.example.com || {
        echo "Request failed, will retry" >&2
        exit 1
      }

tasks:
  main:
    flow:
      fetch:
        action: call-api
        retry:
          max_attempts: 3
          delay: "10s"
```

### 6. Monitor retry metrics

In the UI:
- View retry attempt count and history on step detail
- Check `retry_of_job_id` on task-retried jobs to trace failure chains
- Use server logs to identify patterns (e.g., if step fails on Attempt 1 every time, increase initial delay)

## Examples

### Example 1: Flaky external API

```yaml
actions:
  fetch-weather:
    type: script
    script: |
      curl -f "https://api.weather.example.com/forecast?lat=55&lon=12" || exit 1

tasks:
  get-forecast:
    input:
      city: { type: string }
    flow:
      weather:
        action: fetch-weather
        retry:
          max_attempts: 5
          delay: "5s"
          backoff: exponential
          jitter: true

      process:
        action: process-forecast
        depends_on: [weather]
        input:
          data: "{{ weather.output }}"
```

Calls the weather API with exponential backoff (5s, 10s, 20s, 40s, 80s) and jitter.

### Example 2: Database migration retry

```yaml
tasks:
  run-migration:
    retry:
      max_attempts: 2
      delay: "30s"
    flow:
      check-db:
        action: health-check-db

      migrate:
        action: run-sql-migration
        depends_on: [check-db]

      verify:
        action: verify-schema
        depends_on: [migrate]

    on_success:
      - action: notify-team
        input:
          message: "Migration {{ hook.task_name }} completed"

    on_error:
      - action: alert-dba
        input:
          message: "Migration failed: {{ hook.error_message }}"
```

Entire migration retries if any step fails, up to 2 total attempts.

### Example 3: Batch processing with step retries

```yaml
tasks:
  process-batch:
    input:
      items: { type: array }
    flow:
      fetch-items:
        action: get-items-from-db
        input:
          filter: "{{ input.filter }}"

      process:
        action: transform-item
        depends_on: [fetch-items]
        for_each: "{{ fetch_items.output.items | json_encode() }}"
        retry:
          max_attempts: 3
          delay: "10s"
          backoff: exponential
        continue_on_failure: true   # an exhausted item doesn't fail the job
        input:
          item: "{{ each.item }}"

      aggregate:
        action: combine-results
        depends_on:
          - step: process
            accept: [completed, failed]
        input:
          results: "{{ process.output }}"
```

Each item is processed with step retry. If any item exhausts its retries, the loop's placeholder rolls up `failed` regardless — `continue_on_failure` on `process` itself only decides whether that failure fails the *job*, not the rollup status. `aggregate`'s own `accept: [completed, failed]` is what lets it run and see the partial array, with `null` for any exhausted item; without it, an exhausted item would skip `aggregate` `unreachable` even with `continue_on_failure` set.

## Troubleshooting

### Steps keep retrying, slowing down the job

**Symptom:** A step retries repeatedly, causing the job to stall.

**Causes:**
- `max_attempts` too high (e.g., 10 with 30s delay = 5 minutes of retrying)
- `delay` too long or `backoff` too aggressive

**Fix:**
- Lower `max_attempts` to 2–3
- Use `fixed` backoff instead of exponential
- Investigate root cause — if failure is consistent, retries won't help

### Hooks fire on each retry attempt

**Symptom:** `on_error` hook fires multiple times during retries.

**Cause:** Misunderstanding — hooks only fire when the step/task is **terminal** (all retries exhausted).

**Verify:** Check hook logs; they should appear once, after the final attempt.

### Task retry creates orphaned jobs

**Symptom:** Retry chain breaks in the UI.

**Cause:** This shouldn't happen; `retry_of_job_id` foreign key ensures referential integrity.

**Fix:** Check logs for errors during retry creation. Contact support if the issue persists.

### Exponential backoff delays feel wrong

**Symptom:** Delays don't seem to match expected exponential growth.

**Cause:** Backoff is capped at 64x the initial delay. With `delay: 10s`, max is 640s (~11 min).

**Verify:** Check the step detail UI, which shows the delay before each retry.

---

See the [Workflow YAML reference](https://fremvaerk.github.io/stroem/reference/workflow-yaml/#retry) for complete field documentation.

---

## Triggers

Triggers define automated task execution. Two types are supported: `scheduler` (cron-based) and `webhook` (HTTP-triggered).

:::note[Disabling all triggers of a workspace on one server]
Every trigger has its own `enabled` flag (in the workspace YAML, so it applies wherever the repository is loaded). To stop a whole workspace's triggers on a single server (for example a staging server that mirrors production), set `triggers: false` on that workspace in the **server** config instead — see [Multi-Workspace → Disabling triggers per server](https://fremvaerk.github.io/stroem/guides/multi-workspace/#disabling-triggers-per-server). Cron, webhook, and event-source triggers are all affected; tasks stay runnable manually.
:::

## Cron scheduler

```yaml
triggers:
  nightly-backup:
    type: scheduler
    cron: "0 0 2 * * *"
    task: backup-db
    input:
      retention_days: 30
    enabled: true
```

### Cron expressions

The `cron` field supports both standard 5-field (minute granularity) and extended 6-field (second granularity) expressions:

```
# 5-field: minute hour day-of-month month day-of-week
cron: "0 2 * * *"           # Every day at 2:00 AM

# 6-field: second minute hour day-of-month month day-of-week
cron: "0 0 2 * * *"         # Every day at 2:00:00 AM
cron: "*/10 * * * * *"      # Every 10 seconds
cron: "0 30 9 * * MON-FRI"  # Weekdays at 9:30 AM
```

Extended features (via the `croner` library):
- `L` — last day of month or last weekday occurrence (`5L` = last Friday)
- `#` — nth weekday (`5#2` = second Friday of the month)
- `W` — closest weekday to a day (`15W` = closest weekday to the 15th)
- Text names — `MON`, `TUE`, `JAN`, `FEB`, etc.

### How the scheduler works

- The scheduler runs inside the server process and wakes only when a trigger is due (no fixed polling interval).
- When workspace configs are hot-reloaded, the scheduler picks up new/changed/removed triggers automatically.
- If a trigger's cron expression changes, its next fire time is recalculated. If unchanged, the existing schedule is preserved.
- Jobs created by triggers have `source_type: "trigger"` and `source_id: "{workspace}/{trigger_name}"` for audit trail.
- If the server was down when a trigger was due, it fires on the next startup.

A trigger's `task` may name another workspace's task (`ws.task`), and may
carry a `ref` — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/). Such a trigger creates its
job in the task's owner workspace (at that commit, with `ref`). The ref is
resolved when the trigger fires, before the concurrency policy; a fire that
cannot resolve its task or ref is logged as `Trigger '…' MISSED: …` and has
no other effect (no `cancel_previous`, no skipped row). The concurrency
policy and `source_id` stay keyed on the **defining** workspace's trigger,
and `triggers: false` on the defining workspace still suppresses it.

### Last run and missed runs

Every trigger shows its **last run**. A scheduler trigger can also show **missed runs**. Both appear on the workspace page, on the task page and on the Dashboard.

**Last run.** A trigger's runs are the jobs it created: `source_type: "trigger"` for a scheduler, `"webhook"` for a webhook. Jobs that someone started by hand are not runs of the trigger. This includes a manual run of the same task, a Re-run and a Restart.

- The time links to the job. The result is the job's status: `pending`, `running`, `completed`, `failed`, `cancelled` or `skipped`.
- If the run has [task retry](https://fremvaerk.github.io/stroem/guides/retry/) jobs, the result is the status of the newest retry job. A nightly run that failed once and passed on retry shows `completed`. The time still links to the trigger's own run.
- With `concurrency: skip`, every fire that finds the previous job still running writes a `skipped` row. The last run ignores those rows. Only a trigger with no other run shows a `skipped` last run.
- `never` means the trigger has no run in the database.
- `no access` means the run exists, but you may not view that job. The job is checked on its own task and folder, so a cross-workspace `ws.task` target or a pinned `ref:` job can be hidden even when the trigger is visible.

**Missed runs.** For an enabled scheduler trigger, a run is missed when a scheduled fire time:

- is later than the trigger's newest job of any status (a `skipped` row counts, because it proves the scheduler fired), and
- is at least 5 minutes in the past, and
- has no job.

The UI shows the number of missed runs and the first missed time. The count stops at 1000 and shows `1000+`. The 5 minute grace period lets a fire that is late by seconds pass. A fire recorded less than 1 second before the next scheduled time is taken as that time's own fire, because the database clock can run slightly behind the scheduler's clock. If it was instead a very late fire of the previous scheduled time, that next time is not counted. At most one scheduled time is skipped this way. A trigger with no job at all never shows a mark, because the server cannot know when the trigger was added. A disabled trigger and a trigger in a workspace with `triggers: false` never show a mark. Webhook and event source triggers have no schedule, so they show a last run but never a missed mark.

The mark clears only when the trigger itself runs again. A manual run, a Re-run or a Restart of the task does not clear it, and does not change the last result.

**Where it shows.**

- Workspace page: a "Last run" column in the Triggers table. A warning box above the table lists the triggers with missed runs.
- Task page: the same last run and missed mark on each trigger of the task.
- Dashboard: a card "Triggers that need attention". It lists triggers that have missed runs or whose last result is `failed`, missed first. It hides when no trigger needs attention, and when the request fails. `cancelled` and `skipped` results do not count. Disabled triggers never appear. API: `GET /api/triggers/attention`.

A trigger appears only if its task is not denied for you by [ACL](https://fremvaerk.github.io/stroem/operations/authorization/). If the server cannot load your ACL data, `GET /api/workspaces/{ws}/triggers` now returns an error. Before, it listed every trigger.

**Limitations.**

- The missed rule uses the trigger's current cron and timezone. After you change either, the mark can be wrong until the trigger fires once on the new schedule.
- A trigger that you re-enable after `enabled: false` (or a workspace that you switch back from `triggers: false`) shows the fire times it skipped as missed, until it fires again. The database does not record when it was enabled.
- Job retention (`retention.job_days`) can delete the last run of a rarely firing trigger. The trigger then shows `never` and no mark.
- A trigger that you rename in YAML is a new trigger. It shows `never` until it fires.
- The Dashboard does not list triggers of a workspace that is in load error. They appear again after the workspace loads, with the runs that were missed during the outage.
- A failed run that you already handled by hand still shows as failed until the trigger runs again.

Run status needs two indexes on `job`. See [Migration 053](https://fremvaerk.github.io/stroem/operations/migration-053/) for the manual pre-run on a large database.

### Concurrency policy

Control what happens when a trigger fires while a previous run is still active:

```yaml
triggers:
  hourly-etl:
    type: scheduler
    cron: "0 * * * *"
    task: etl-pipeline
    concurrency: skip        # skip | allow | cancel_previous
```

| Policy | Behavior |
|--------|----------|
| `allow` | (default) Always create a new job, even if previous runs are still active. |
| `skip` | Skip the trigger if there is already an active job from this trigger. Creates a job record with `status: skipped` for visibility (no steps are executed). |
| `cancel_previous` | Cancel all active jobs from this trigger, then create a new job. |

The concurrency check uses the trigger's `source_type` and `source_id` (`"{workspace}/{trigger_name}"`) to identify related jobs.

### Timezone support

Scheduler triggers support optional timezone specification to run cron expressions in a specific timezone rather than UTC:

```yaml
triggers:
  nightly-build:
    type: scheduler
    cron: "0 2 * * *"
    timezone: "Europe/Copenhagen"    # Runs at 2:00 AM Copenhagen time
    task: build
```

- **`timezone` field**: Optional IANA timezone name (e.g., `"America/New_York"`, `"Australia/Sydney"`, `"Europe/London"`)
- **Default**: UTC when not specified
- **DST handling**: Daylight Saving Time transitions are handled automatically:
  - Spring-forward gaps (time jumps ahead): the trigger fires at the first valid time after the gap
  - Fall-back overlaps (time repeats): the trigger fires once at the first occurrence
- **Timezone names**: Use standard IANA names from the [IANA Time Zone Database](https://www.iana.org/time-zones) (case-sensitive)

Common timezone examples:
- `"America/New_York"` — Eastern Time (ET)
- `"America/Los_Angeles"` — Pacific Time (PT)
- `"Europe/London"` — Greenwich Mean Time (GMT)
- `"Europe/Paris"` — Central European Time (CET)
- `"Asia/Tokyo"` — Japan Standard Time (JST)
- `"Australia/Sydney"` — Australian Eastern Time (AEST)

### Scheduler fields

| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | `scheduler` |
| `cron` | Yes | Cron expression (5 or 6 fields) |
| `task` | Yes | Name of the task to execute. May be another workspace's task (`ws.task`) |
| `ref` | No | Branch, tag or full commit SHA of the task's owner workspace to run the task from — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/) |
| `input` | No | Input values passed to the task |
| `timezone` | No | IANA timezone name (default: `"UTC"`). Example: `"Europe/Copenhagen"` |
| `concurrency` | No | What to do when previous runs are active: `allow` (default), `skip`, `cancel_previous` |
| `enabled` | No | Whether the trigger is active (default: `true`) |

## Webhook triggers

Webhook triggers expose an HTTP endpoint that external systems (GitHub, GitLab, monitoring tools) can call to trigger tasks.

```yaml
triggers:
  on-push:
    type: webhook
    name: github-push          # URL-safe name — endpoint: POST /hooks/github-push
    task: ci-pipeline
    secret: "whsec_abc123"     # Optional — omit for public webhooks
    input:                     # Optional — default values merged into request input
      environment: staging
    enabled: true

  deploy-sync:
    type: webhook
    name: deploy
    task: do-deploy
    mode: sync                 # Wait for job completion before responding
    timeout_secs: 60           # Max wait time (default: 30, max: 300)
    secret: "whsec_deploy"
```

### Calling a webhook

```bash
# POST with JSON body and secret via query param
curl -X POST http://localhost:8080/hooks/github-push?secret=whsec_abc123 \
  -H "Content-Type: application/json" \
  -d '{"ref": "refs/heads/main", "commits": []}'

# GET with secret via Authorization header
curl http://localhost:8080/hooks/github-push \
  -H "Authorization: Bearer whsec_abc123"

# Public webhook (no secret configured) — no auth needed
curl -X POST http://localhost:8080/hooks/public-hook \
  -H "Content-Type: application/json" \
  -d '{"event": "deploy"}'
```

### Authentication

- If the trigger has a `secret` field, callers must provide it via `?secret=xxx` query parameter or `Authorization: Bearer xxx` header.
- If no `secret` is configured, the webhook is public.
- Invalid or missing secrets return `401 Unauthorized`.
- With `force_refresh: true`, the caller is authenticated against the
  definition the server has loaded **before** the workspace is reloaded, so
  an unauthenticated caller can never trigger a git fetch. After the reload
  the webhook is matched again (gone → `404`) and the caller is
  authenticated again against the refreshed `secret`; the job is created
  from the refreshed definition, so a changed `task` or `ref` takes effect.
  A secret that was rotated in git is therefore accepted only once the
  server has loaded it (its watcher poll, or an earlier refresh). A reload
  that leaves the workspace unavailable answers `500`.
- An unknown task, a bad or missing `ref` and similar configuration errors
  answer `400`; a transient problem (for example, the git server is
  unreachable) answers `500`.

### Input structure

The webhook handler wraps the entire HTTP request into a structured input map:

```json
{
  "body": { "ref": "refs/heads/main", "commits": [] },
  "headers": { "content-type": "application/json", "x-github-event": "push" },
  "method": "POST",
  "query": { "env": "production" },
  "environment": "staging"
}
```

- `body`: JSON-parsed if `Content-Type: application/json`, raw string otherwise, `null` for GET
- `headers`: lowercase key map of all request headers
- `method`: `"GET"` or `"POST"`
- `query`: query parameters (the `secret` param is excluded)
- Trigger YAML `input` defaults merge at top level (don't overwrite `body`, `headers`, `method`, `query`)

### Webhook fields

| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | `webhook` |
| `name` | Yes | URL-safe name (alphanumeric, hyphens, underscores) |
| `task` | Yes | Name of the task to execute. May be another workspace's task (`ws.task`) |
| `ref` | No | Branch, tag or full commit SHA of the task's owner workspace to run the task from — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/) |
| `secret` | No | Secret for authentication |
| `input` | No | Default input values merged with request data |
| `enabled` | No | Whether the trigger is active (default: `true`) |
| `mode` | No | `"async"` (default) or `"sync"` — sync waits for job completion |
| `timeout_secs` | No | Max wait time in sync mode (default: 30, max: 300) |

### Sync vs async mode

By default, webhooks return immediately with a `job_id` (async mode). The caller must poll `GET /api/jobs/{id}` to track completion.

In **sync mode** (`mode: sync`), the webhook handler blocks until the job reaches a terminal state (completed or failed) or the timeout is reached.

**Sync response** (job completed or failed):
```json
{
  "job_id": "...",
  "trigger": "deploy",
  "task": "do-deploy",
  "status": "completed",
  "output": { "result": "ok" }
}
```
HTTP status: `200 OK`.

**Timeout response** (job still running):
```json
{
  "job_id": "...",
  "trigger": "deploy",
  "task": "do-deploy",
  "status": "running"
}
```
HTTP status: `202 Accepted` — use the `job_id` to poll manually.

The `output` of a sync response, and of the job-status poll below, is
masked like job detail: workspace secrets (and the secrets of any
[pinned commit](https://fremvaerk.github.io/stroem/guides/git-refs/#secrets-and-redaction) the job is connected
to) appear as `••••••`. If those secrets cannot be loaded at the moment, the
answer is `503` with `{"job_id": "...", "error": "redaction set unavailable,
retry"}` — the job exists; poll it again later.

### Checking async job status

When using async webhooks, the response includes a `job_id`. You can check the job status using the same webhook endpoint — no API key or JWT required:

```bash
# Trigger the webhook (async)
curl -X POST http://localhost:8080/hooks/my-webhook?secret=whsec_abc123 \
  -H "Content-Type: application/json" \
  -d '{"ref": "main"}'
# Response: {"job_id":"abc-123","trigger":"my-webhook","task":"deploy"}

# Check job status (same secret)
curl http://localhost:8080/hooks/my-webhook/jobs/abc-123?secret=whsec_abc123
# Response: {"job_id":"abc-123","trigger":"my-webhook","task":"deploy","status":"running","output":null}

# Wait for completion (blocks until done or timeout)
curl "http://localhost:8080/hooks/my-webhook/jobs/abc-123?secret=whsec_abc123&wait=true&timeout=60"
# Response: {"job_id":"abc-123","trigger":"my-webhook","task":"deploy","status":"completed","output":{...}}
```

The status endpoint only returns jobs created by that specific webhook trigger — it cannot be used to query arbitrary jobs.

See [Webhook API](https://fremvaerk.github.io/stroem/reference/webhook-api/) for the full endpoint reference.

### Webhook name uniqueness

Webhook names should be unique across all workspaces. If the same name appears in multiple workspaces, the first match wins at dispatch time.

### Last run

A webhook trigger shows its last run (the newest job it created, with the result) on the workspace page, the task page and the Dashboard. It never shows a missed mark, because a webhook has no schedule. See [Last run and missed runs](#last-run-and-missed-runs).

## Event source triggers

For long-running queue consumers that create jobs from external events, see the dedicated [Event Sources](https://fremvaerk.github.io/stroem/guides/event-sources/) guide.

Event sources reference a consumer task and a target task:

```yaml
triggers:
  sqs-events:
    type: event_source
    task: sqs-consumer         # consumer task (runs continuously)
    target_task: process-order # where OUTPUT: lines create jobs
    restart_policy: always
    max_in_flight: 10
```

The consumer task runs as a regular job. Each `OUTPUT: ` line emitted by the consumer is parsed as JSON and creates a new job for the target task. This pattern is ideal for queue-driven architectures where you need continuous consumption with backpressure support.

An event source trigger shows a last run too, but never a missed mark. Its last run is the newest **consumer** job, not a job that the consumer emitted. The server finds consumer jobs by the `_fingerprint` marker in the job input. A running consumer shows "running since" its start time. See [Last run and missed runs](#last-run-and-missed-runs).

---

## Event Source Triggers

Event source triggers enable integration with external message queues and event streams. An event source is a long-running process that consumes messages from a queue (SQS, Kafka, RabbitMQ, Pub/Sub, etc.) and creates jobs by emitting `OUTPUT: <json>` lines to stdout.

Strøm is **queue-agnostic** — you bring your own consumer logic. The event source trigger references a task that runs the consumer process. When the consumer emits lines to stdout using the `OUTPUT: ` prefix protocol, Strøm creates jobs for the target task. Other stdout and stderr output is captured as log entries in the consumer's logs.

## Overview

Event sources differ from other trigger types:

| Feature | Scheduler | Webhook | Event Source |
|---------|-----------|---------|--------------|
| Trigger method | Cron schedule | HTTP POST | Long-running process (OUTPUT: prefix) |
| Execution | Periodic | On-demand | Continuous (with restart policy) |
| Duration | Minutes to hours | On-demand | Hours/days (permanent fixture) |
| Use case | Backups, reports, ETL | Deployments, CI/CD | Queue consumption, streaming |

## Configuration

Event sources are defined in the `triggers:` section of your workspace YAML. The event source trigger references a **consumer task** (a regular task whose flow runs the long-lived consumer process) and specifies a **target task** where jobs are created from emitted events:

```yaml
actions:
  sqs-poll:
    type: script
    language: python
    dependencies: ["boto3"]
    script: |
      # Your queue consumer code here
      # Emit events via: print("OUTPUT: " + json.dumps(event))

tasks:
  sqs-consumer:
    flow:
      poll:
        action: sqs-poll

triggers:
  sqs-events:
    type: event_source
    task: sqs-consumer              # Consumer task (runs the long-lived process)
    target_task: process-order      # Target task for emitted jobs
    env:                            # Tera-templated environment variables
      QUEUE_URL: "{{ secret.queue_url }}"
    input:                          # Default input merged into each job
      priority: high
    restart_policy: always          # always, on_failure, never
    backoff_secs: 5                 # Initial restart delay (exponential backoff)
    max_in_flight: 10               # Backpressure: pause reading when limit reached
    enabled: true
```

**Key differences from regular actions:**

- **`task:`** specifies the consumer task (contains the flow with your consumer action)
- **`target_task:`** specifies which task to create jobs for (receives the emitted JSON as input)
- **No inline execution fields** on the trigger — the consumer's execution is defined in the referenced task
- **`env:`** provides environment overrides for the consumer execution
- **`input:`** provides defaults merged into each emitted job

## Stdout protocol

Event source scripts communicate with the worker using a line-based protocol over stdout:

- Lines beginning with `OUTPUT: ` (note the space after the colon) followed by a JSON object trigger a job.
- All other stdout lines are treated as regular log output and appear in the event source job's logs.
- Empty lines are ignored.

**Example:** A consumer reading from an SQS queue might emit:

```
Starting consumer for queue: https://sqs.eu-west-1.amazonaws.com/123456789/orders
Connected successfully. Waiting for messages...
OUTPUT: {"orderId": "123", "amount": 99.99, "customer": "alice"}
Received message, deleting from queue...
OUTPUT: {"orderId": "124", "amount": 150.00, "customer": "bob"}
OUTPUT: {"orderId": "125", "amount": 45.50, "customer": "charlie"}
```

Each `OUTPUT: ` line becomes a separate job, with the JSON object merged into the task's input. Fields from the trigger's `input` section are defaults; JSON fields override them. Regular text lines appear in the worker's job log so you can monitor consumer activity.

### Why OUTPUT: prefix?

This protocol is consistent with how regular script actions work in Strøm. It lets you:

- Emit regular log output without it being misinterpreted as a job trigger
- Mix diagnostic messages, status updates, and event emissions freely in a single script
- Use `print()` / `echo` for debugging without side effects

### Parsing and errors

- **`OUTPUT: <json>`**: JSON parsed and merged with default input to create a job
- **Other non-empty lines**: Treated as log output; appear in job logs
- **Empty lines**: Ignored
- **Malformed JSON after `OUTPUT: `**: Logged as a warning; line skipped; event source continues

This design ensures robust operation — a single malformed message doesn't crash the consumer.

## Runner examples

### Local script with Python and SQS

```yaml
actions:
  sqs-poll:
    type: script
    language: python
    dependencies: ["boto3"]
    script: |
      import boto3
      import json
      import sys
      import os

      sqs = boto3.client("sqs")
      queue_url = os.environ["QUEUE_URL"]

      while True:
          resp = sqs.receive_message(
              QueueUrl=queue_url,
              WaitTimeSeconds=20,
              MaxNumberOfMessages=1
          )
          for msg in resp.get("Messages", []):
              # Parse and emit using OUTPUT: prefix
              body = json.loads(msg["Body"])
              print("OUTPUT: " + json.dumps(body))
              sys.stdout.flush()
              # Delete from queue
              sqs.delete_message(
                  QueueUrl=queue_url,
                  ReceiptHandle=msg["ReceiptHandle"]
              )

tasks:
  sqs-consumer:
    flow:
      poll:
        action: sqs-poll

  process-order:
    mode: distributed
    input:
      orderId: { type: string }
      amount: { type: number }
    flow:
      validate:
        action: validate-order
        input:
          order_id: "{{ input.orderId }}"
      process:
        action: charge-and-ship
        depends_on: [validate]
        input:
          order_id: "{{ input.orderId }}"
          amount: "{{ input.amount }}"

triggers:
  sqs-events:
    type: event_source
    task: sqs-consumer
    target_task: process-order
    env:
      QUEUE_URL: "{{ secret.sqs_queue_url }}"
      AWS_REGION: "eu-west-1"
    restart_policy: always
    backoff_secs: 5
    max_in_flight: 10
```

### Docker container with Kafka

```yaml
actions:
  kafka-poll:
    type: script
    runner: docker
    image: myorg/kafka-consumer:1.0
    # No script field — the image runs as-is

tasks:
  kafka-consumer:
    flow:
      poll:
        action: kafka-poll

  handle-event:
    flow:
      process:
        action: process-kafka-event

triggers:
  kafka-events:
    type: event_source
    task: kafka-consumer
    target_task: handle-event
    env:
      KAFKA_BROKERS: "{{ secret.kafka_brokers }}"
      KAFKA_TOPIC: "events"
      KAFKA_GROUP: "stroem-worker-1"
    input:
      source: kafka
    restart_policy: always
    max_in_flight: 20
    backoff_secs: 10
```

Your Docker image should accept configuration via environment variables and emit JSON lines to stdout. Example Dockerfile:

```dockerfile
FROM golang:1.21-alpine
WORKDIR /app
COPY . .
RUN go build -o consumer .
CMD ["./consumer"]
```

And your consumer code:

```go
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    "strings"

    "github.com/segmentio/kafka-go"
)

func main() {
    brokers := strings.Split(os.Getenv("KAFKA_BROKERS"), ",")
    topic := os.Getenv("KAFKA_TOPIC")
    group := os.Getenv("KAFKA_GROUP")

    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers: brokers,
        Topic:   topic,
        GroupID: group,
    })

    for {
        msg, _ := reader.ReadMessage(context.Background())
        var event map[string]interface{}
        json.Unmarshal(msg.Value, &event)
        output, _ := json.Marshal(event)
        fmt.Println("OUTPUT: " + string(output))
    }
}
```

### Kubernetes pod with RabbitMQ

```yaml
actions:
  rabbitmq-poll:
    type: script
    runner: pod
    image: myorg/amqp-consumer:latest
    manifest:
      metadata:
        labels:
          app: stroem-rabbitmq-consumer
      spec:
        serviceAccountName: stroem-worker
        imagePullSecrets:
          - name: myorg-registry
        containers:
          - name: main
            resources:
              requests:
                memory: "256Mi"
                cpu: "250m"
              limits:
                memory: "512Mi"
                cpu: "500m"

tasks:
  rabbitmq-consumer:
    flow:
      poll:
        action: rabbitmq-poll

  process-message:
    flow:
      handle:
        action: handle-message

triggers:
  rabbitmq-events:
    type: event_source
    task: rabbitmq-consumer
    target_task: process-message
    env:
      AMQP_URL: "{{ secret.amqp_url }}"
      QUEUE_NAME: "orders"
    restart_policy: always
    backoff_secs: 5
    max_in_flight: 15
```

## Worker configuration

Event source consumer tasks run through normal runners (local, Docker, or Kubernetes), so no special worker configuration is required. Workers simply need to declare the appropriate runner tags for the actions used in their consumer task:

```yaml
# worker-config.yaml
worker_token: "wk_abc123..."
tags:
  - script          # Required if consumer uses type: script with runner: local
  - docker          # Required if consumer uses runner: docker
  - pod             # Required if consumer uses runner: pod
```

The server-side `EventSourceManager` background task handles consumer lifecycle: creation, restart policy, exponential backoff, and monitoring. Consumers run indefinitely (or until failure, depending on `restart_policy`) as regular jobs executed through the normal claiming mechanism.

## Restart policies

When an event source process exits, the restart policy determines what happens next.

| Policy | Behavior |
|--------|----------|
| `always` (default) | Restart immediately, regardless of exit code. Use for permanent consumers. |
| `on_failure` | Restart only if exit code is non-zero. Stops automatically on clean exit. |
| `never` | Do not restart. Process exits once and the trigger becomes idle. |

### Exponential backoff

When `restart_policy: always` or `on_failure`, restarts use **exponential backoff** to prevent thundering herd:

- First restart: `backoff_secs` seconds (default 5)
- Second consecutive failure: `backoff_secs * 2` (10s)
- Third: `backoff_secs * 4` (20s)
- ... exponential growth until capped at 5 minutes

The counter resets on a successful run (exit code 0).

Example with 10-second base backoff:

```yaml
actions:
  my-consumer:
    type: script
    script: "./my_consumer.sh"

tasks:
  my-task:
    flow:
      consume:
        action: my-consumer

triggers:
  flaky-consumer:
    type: event_source
    task: my-task
    target_task: process-event
    restart_policy: on_failure
    backoff_secs: 10
```

## Backpressure and max_in_flight

The `max_in_flight` setting provides **natural Unix pipe backpressure**. When the specified number of jobs are pending or running:

1. The worker pauses reading from the event source's stdout
2. The pipe fills up, blocking the consumer process
3. Consumer naturally slows down or pauses
4. When jobs complete, the worker resumes reading

This prevents job queues from exploding when downstream tasks slow down.

```yaml
actions:
  consume:
    type: script
    runner: docker
    image: myorg/consumer:latest

tasks:
  high-volume-consumer:
    flow:
      consume:
        action: consume

triggers:
  high-volume-stream:
    type: event_source
    task: high-volume-consumer
    target_task: process-event
    max_in_flight: 50  # Never create more than 50 pending/running jobs at once
```

Without `max_in_flight`, all events are immediately converted to jobs, potentially overwhelming the system.

## How it works internally

### Job creation flow

1. **Consumer task runs**: The server creates a job for the consumer task, which runs indefinitely
2. **Consumer emits events**: The consumer process writes `OUTPUT: {"field": "value"}` to stdout
3. **Events are captured**: The consumer's log streaming captures `OUTPUT: ` lines
4. **Jobs created for target**: When events are emitted via `POST /worker/event-source/emit`, they are parsed and merged with trigger `input` defaults
5. **Target task invoked**: A new job is created for the `target_task`, with the merged JSON as input, `source_type: "event_source"`, `source_id: "{workspace}/{trigger_name}"`
6. **Backpressure tracked**: The server maintains a count of pending/running jobs. When `max_in_flight` is reached, the worker pauses stdout reading, naturally slowing the consumer

### EventSourceManager

The server runs an `EventSourceManager` background task that:

- Periodically checks for event source triggers in workspace configs
- Creates and monitors consumer task jobs
- Restarts consumer jobs according to `restart_policy` and exponential backoff
- Handles consumer lifecycle (start, failure, restart with delay)

Consumer jobs are regular jobs created via the normal task dispatch mechanism, so they respect timeouts, recovery sweeps, and all standard job features.

### Error handling

- **Consumer crashes**: Handled by `restart_policy` + exponential backoff
- **Malformed JSON**: Logged; event skipped; consumer continues
- **Network errors**: Depends on consumer retry logic (your code)
- **Worker crashes**: EventSourceManager detects stale controller, restarts it
- **Server down**: Consumer keeps running; jobs are created when server comes back up

## Disabling event sources

Set `enabled: false` to pause an event source without removing it:

```yaml
triggers:
  my-consumer:
    type: event_source
    task: my-task
    target_task: process-event
    enabled: false  # Process won't start
```

An event-source trigger cannot carry a [`ref:`](https://fremvaerk.github.io/stroem/guides/git-refs/) yet:
`stroem validate` reports it, and the server does not start such a consumer
(it logs a warning on every reconcile).

## Secrets and templating

Event variables and `input` defaults support Tera templating, with access to `secret.*`:

```yaml
triggers:
  secure-consumer:
    type: event_source
    task: protected-task
    script: |
      #!/bin/bash
      # Will have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from env
      aws sqs receive-message --queue-url "$QUEUE_URL" | jq '.Messages[].Body | fromjson'
    env:
      QUEUE_URL: "{{ secret.queue_url }}"
      AWS_ACCESS_KEY_ID: "{{ secret.aws_access_key }}"
      AWS_SECRET_ACCESS_KEY: "{{ secret.aws_secret_key }}"
    input:
      region: "{{ secret.default_region }}"
```

Secrets are injected at runtime; never logged or exposed in job configs.

## Best practices

- **Idempotent consumption**: Emit an event ID (`job_id`) in each JSON line; task can check for duplicates
- **Structured input**: Emit consistent JSON schema; document expected fields
- **Graceful shutdown**: Handle `SIGTERM` to finish in-flight work before exiting
- **Health checks**: Include a heartbeat or status endpoint your alerting can monitor
- **Error logging**: Write errors and diagnostics to stderr; they appear in the event source job's log view
- **Resource limits**: Set `resources:` in pod `manifest` to prevent runaway consumers
- **Rate limiting**: Control burst via `max_in_flight`; use `sequential: true` in task flow for strict ordering

## Example: Multi-message batch consumer

Process multiple messages per event source "tick":

```yaml
actions:
  sqs-batch-poll:
    type: script
    language: python
    dependencies: ["boto3"]
    script: |
      import boto3
      import json
      import os

      sqs = boto3.client("sqs")
      queue_url = os.environ["QUEUE_URL"]

      while True:
          # Receive up to 10 messages at once
          resp = sqs.receive_message(
              QueueUrl=queue_url,
              MaxNumberOfMessages=10,
              WaitTimeSeconds=20
          )

          messages = resp.get("Messages", [])
          if not messages:
              continue

          # Emit a single event with batch of messages using OUTPUT: prefix
          batch = [json.loads(msg["Body"]) for msg in messages]
          print("OUTPUT: " + json.dumps({"messages": batch, "count": len(batch)}))

          # Clean up
          for msg in messages:
              sqs.delete_message(
                  QueueUrl=queue_url,
                  ReceiptHandle=msg["ReceiptHandle"]
              )

tasks:
  batch-consumer:
    flow:
      poll:
        action: sqs-batch-poll

  process-batch:
    mode: distributed
    input:
      messages: { type: array }
      count: { type: integer }
    flow:
      iterate:
        action: process-one-batch
        input:
          batch: "{{ input.messages }}"

triggers:
  batch-processor:
    type: event_source
    task: batch-consumer
    target_task: process-batch
    env:
      QUEUE_URL: "{{ secret.sqs_queue_url }}"
    max_in_flight: 5  # Limit concurrent batches
```

## Example: Multi-step consumer

A consumer task can have multiple steps — setup actions, cleanup tasks, or complex flows. The consumer keeps running as long as one step in the flow remains active:

```yaml
actions:
  create-consumer-group:
    type: script
    language: bash
    script: |
      #!/bin/bash
      set -e
      export KAFKA_BROKERS="${KAFKA_BROKERS}"
      kafka-consumer-groups --bootstrap-server "$KAFKA_BROKERS" \
        --create --group "$CONSUMER_GROUP" || true  # Ignore if exists

  poll-kafka:
    type: docker
    image: myorg/kafka-poller:latest

  cleanup-on-exit:
    type: script
    language: bash
    script: |
      #!/bin/bash
      echo "Consumer shutdown, cleaning up resources..."

tasks:
  kafka-consumer:
    flow:
      setup:
        action: create-consumer-group
      consume:
        action: poll-kafka
        depends_on: [setup]
    on_success:
      - action: cleanup-on-exit
    on_error:
      - action: cleanup-on-exit
    on_cancel:
      - action: cleanup-on-exit

  process-kafka-event:
    flow:
      process:
        action: handle-kafka-message
        input:
          message: "{{ input }}"

triggers:
  kafka-stream:
    type: event_source
    task: kafka-consumer
    target_task: process-kafka-event
    env:
      KAFKA_BROKERS: "{{ secret.kafka_brokers }}"
      KAFKA_TOPIC: "events"
      CONSUMER_GROUP: "stroem-worker-1"
    restart_policy: always
    backoff_secs: 5
    max_in_flight: 20
```

The `consume` step runs indefinitely while emitting `OUTPUT: ` lines. A dependent's own `accept: [failed]` (or `accept: any`) would now let it run even after `consume` fails — but that's a `depends_on` question, separate from `continue_on_failure` on `consume` itself, which also changes how the consumer's own failure interacts with `restart_policy` (see [Restart policies](#restart-policies) above). Neither is what a cleanup step needs here: `cleanup-on-exit` runs as an `on_success` / `on_error` / `on_cancel` hook instead, so it always fires exactly once when the job reaches a terminal state, whatever that state is and however `consume` got there, independent of the restart policy that separately determines whether the consumer task restarts.

## Comparison with other triggers

**Cron scheduler**: Great for scheduled, periodic work (backups, reports). Fixed timing, simple configuration. Not suitable for reacting to external events.

**Webhook**: Excellent for integrations (GitHub, Slack, deployments). External system initiates the request. Real-time but requires inbound network access.

**Event source**: Ideal for queue-driven architecture. Decouples producers from consumers. Self-managed consumer lifecycle. Supports backpressure and rate limiting.

Choose event sources when:
- You have a dedicated message queue (SQS, Kafka, RabbitMQ, etc.)
- You need continuous consumption (not periodic)
- Ordering or batching matters
- You want to manage your own consumer logic

## Troubleshooting

**Consumer task not starting?**
- Check that the `task:` field references an existing task in the same workspace
- Verify the task contains at least one flow step with a valid action
- Check worker has the required tags for the runner used in the consumer action
- Verify `enabled: true` on the trigger

**Events not being processed?**
- Verify consumer is emitting lines with the `OUTPUT: ` prefix followed by valid JSON
- Check the consumer task's job logs — look for any errors or unexpected output
- Verify `target_task:` references an existing task
- Check `max_in_flight` — if the limit is reached, event emission will block

**Consumer keeps restarting?**
- Check the consumer task job logs for error messages (exit code, exceptions, etc.)
- Verify environment variables are set correctly (check with `env` debugging in consumer)
- Consider increasing `backoff_secs` to allow recovery time between restarts
- Check resource limits aren't being hit (memory, CPU, file descriptors)

**Target jobs failing?**
- Check the `input:` defaults on the trigger — verify required fields are present
- Verify the emitted JSON contains all required input fields for the target task
- Check the target task's action for validation or processing errors

**High latency between event and job creation?**
- Increase `max_in_flight` to allow more parallelism
- Set `sequential: false` in target task flow (default)
- Consider batching events in the consumer if appropriate
- Monitor `max_in_flight` — if consistently full, consumer is waiting for jobs to complete

---

## Hooks

Tasks can define `on_success`, `on_error`, and `on_cancel` hooks that fire automatically when a job reaches a terminal state. Each hook references an existing action and can pass input rendered with job context.

## Syntax

```yaml
tasks:
  deploy:
    flow:
      build:
        action: build-app
      deploy:
        action: deploy-app
        depends_on: [build]
    on_success:
      - action: notify-slack
        input:
          webhook_url: "{{ secret.SLACK_WEBHOOK_URL }}"
          message: "Deploy {{ hook.task_name }} succeeded ({{ hook.duration_secs }}s)"
    on_error:
      - action: notify-slack
        input:
          webhook_url: "{{ secret.SLACK_WEBHOOK_URL }}"
          message: "Deploy {{ hook.task_name }} FAILED: {{ hook.error_message }}"
```

## Hook context variables

Hook input values are Tera templates with access to a `hook` object containing job context, plus workspace `secrets` under the `secret` key:

| Variable | Type | Description |
|----------|------|-------------|
| `hook.workspace` | string | Workspace name |
| `hook.task_name` | string | Task that completed/failed |
| `hook.job_id` | string | UUID of the original job |
| `hook.status` | string | `"completed"`, `"failed"`, or `"cancelled"` |
| `hook.is_success` | bool | true if completed |
| `hook.error_message` | string/null | All failed step errors combined |
| `hook.source_type` | string | Original job source (`"api"`, `"trigger"`, etc.) |
| `hook.source_id` | string/null | Original job source ID |
| `hook.started_at` | string/null | ISO 8601 timestamp |
| `hook.completed_at` | string/null | ISO 8601 timestamp |
| `hook.duration_secs` | number/null | Execution duration in seconds |
| `hook.failed_steps` | array | Failed step details (see below) |
| `hook.revision` | string/null | Workspace revision (git SHA or folder hash) pinned on the job |
| `hook.ref` | string/null | The [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) the job runs at, as written; `null` unless it is a pinned job |

Each entry in `hook.failed_steps` contains:

| Field | Type | Description |
|-------|------|-------------|
| `step_name` | string | Name of the failed step |
| `action_name` | string | Action that was executed |
| `error_message` | string/null | The step's error message |
| `continue_on_failure` | bool | Whether this row's own flow step has `continue_on_failure` set. A loop instance row reports its placeholder's flag |
| `tolerated` | bool | Same value as `continue_on_failure` above — this failure does not by itself fail the job; other untolerated failures or a cancellation can still decide the outcome |
| `carried_over` | bool | `true` if this failure was carried forward from the source run by a job restart rather than produced by this job |

## on_cancel hooks

Tasks can define `on_cancel` hooks that fire when a job is cancelled — whether manually via the API, by a timeout, or by a scheduler concurrency policy (`cancel_previous`).

:::caution[Breaking change]
Previously, cancelled jobs fired `on_error` hooks. Now they fire `on_cancel` only. If you relied on `on_error` to catch cancellations, add an `on_cancel` hook with the same action.
:::

### Syntax

```yaml
tasks:
  deploy:
    flow:
      build:
        action: build-app
      deploy:
        action: deploy-app
        depends_on: [build]
    on_cancel:
      - action: notify-slack
        input:
          webhook_url: "{{ secret.SLACK_WEBHOOK_URL }}"
          message: "Deploy {{ hook.task_name }} was cancelled"
```

`on_cancel` hooks use the same `hook.*` template context as `on_success` / `on_error`, with `hook.status` set to `"cancelled"` and `hook.is_success` set to `false`. If no `on_cancel` hooks are defined, nothing fires for cancelled jobs.

## on_suspended hooks

Tasks can define `on_suspended` hooks that fire when any step in the task enters `suspended` state, typically from an approval gate. Use this to notify approvers or escalate when human action is required.

### Syntax

```yaml
tasks:
  deploy-production:
    on_suspended:
      - action: notify-approvers
        input:
          channel: "#deploy-approvals"
          message: "{{ hook.task_name }} in {{ hook.workspace }} is awaiting approval"
    flow:
      approve:
        type: approval
        message: "Approve production deployment?"
      deploy:
        action: deploy-script
        depends_on: [approve]
```

### Available variables

`on_suspended` hooks get a `hook` object describing the suspended step:

| Variable | Type | Description |
|----------|------|-------------|
| `hook.workspace` | string | Workspace name |
| `hook.task_name` | string | Task containing the suspended step |
| `hook.job_id` | string | UUID of the job |
| `hook.step_name` | string | Name of the suspended approval step |
| `hook.message` | string | Rendered approval message shown to approvers |
| `hook.source_type` | string | Original job source (`"api"`, `"trigger"`, etc.) |
| `hook.source_id` | string/null | Original job source ID |
| `hook.revision` | string/null | Workspace revision (git SHA or folder hash) pinned on the job |
| `hook.ref` | string/null | The [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) the job runs at, as written; `null` unless it is a pinned job |

### Practical examples

**Slack notification with actionable message:**

```yaml
actions:
  notify-slack-approval:
    type: script
    script: |
      curl -X POST "$WEBHOOK_URL" \
        -H 'Content-Type: application/json' \
        -d '{
          "text": "{{ input.message }}",
          "blocks": [
            {
              "type": "section",
              "text": {"type": "mrkdwn", "text": "{{ input.message }}"}
            },
            {
              "type": "context",
              "elements": [{"type": "mrkdwn", "text": "Job ID: {{ input.job_id }}"}]
            }
          ]
        }'
    input:
      message: { type: string }
      job_id: { type: string }

tasks:
  deploy:
    on_suspended:
      - action: notify-slack-approval
        input:
          message: |
            *Approval needed for {{ hook.task_name }}*
            Workspace: {{ hook.workspace }}
            Job: {{ hook.job_id }}
          job_id: "{{ hook.job_id }}"
    flow:
      approve:
        type: approval
        message: "Deploy {{ input.service }} to production?"
      deploy:
        action: deploy-app
        depends_on: [approve]
```

**PagerDuty escalation:**

```yaml
tasks:
  critical-maintenance:
    on_suspended:
      - action: create-pagerduty-incident
        input:
          title: "{{ hook.task_name }} awaiting approval in {{ hook.workspace }}"
          service_id: "{{ secret.pagerduty_service_id }}"
          job_id: "{{ hook.job_id }}"
    flow:
      approve:
        type: approval
        message: "Proceed with critical maintenance?"
        timeout: 1h
      execute:
        action: maintenance-script
        depends_on: [approve]
```

## Behavior

- **Fire-and-forget**: Hook job creation is best-effort. Failures are logged but never affect the original job.
- **No recursion**: Jobs created by hooks (`source_type = "hook"`) never trigger further hooks.
- **Visible as jobs**: Hook jobs appear in the job list with `task_name = "_hook:<action>"` and `source_type = "hook"`. A hook job's detail page links back to the job that fired it (**hook for `<id>`** under the title); over the API that id is the hook job's `source_job_id`.
- **Normal execution**: Hook jobs go through the normal claim/execute flow on workers.
- **Multiple hooks**: You can define multiple hooks per event. They all fire independently.
- **Validation**: Hook action references are validated at parse time — referencing a non-existent action is an error.
- **Failure visibility**: If a hook job fails at runtime, the failure is logged as a server event on the original job (visible in the "Server Events" panel on the job detail page).
- **Pinned jobs**: The hooks of a job that runs on a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) come from that job's commit, and the hook job runs at the same ref and commit.
- **No `ref` on hooks**: `ref:` on a hook, or a hook whose action is a `type: task` action carrying `ref`, is rejected by `stroem validate`. At runtime such a hook is not fired, and the error is logged as a server event on the original job.

## Workspace-level hooks

You can define `on_success`, `on_error`, and `on_cancel` hooks at the top level of your workflow YAML. These act as **fallback defaults**: they fire only when the specific task has no hooks defined for that event type.

```yaml
actions:
  notify-slack:
    type: script
    script: 'curl -X POST "$WEBHOOK_URL" -d "{\"text\": \"$MESSAGE\"}"'
    input:
      message:
        type: string

on_success:
  - action: notify-slack
    input:
      message: "Job {{ hook.task_name }} in {{ hook.workspace }} completed successfully"

on_error:
  - action: notify-slack
    input:
      message: "Job {{ hook.task_name }} in {{ hook.workspace }} FAILED: {{ hook.error_message }}"

tasks:
  deploy:
    flow:
      step1:
        action: deploy-app
    # No on_success/on_error here — workspace-level hooks fire

  backup:
    flow:
      step1:
        action: backup-db
    on_error:
      - action: notify-slack
        input:
          message: "Backup failed — page oncall!"
    # Task-level on_error takes precedence; workspace on_error is skipped
    # Workspace on_success still fires (because task has no on_success)
```

### Rules

- `on_success`, `on_error`, and `on_cancel` are evaluated independently. A task can override one while inheriting the others.
- Workspace hooks only fire for **top-level jobs** (source type `api` — programmatic calls, `user` — authenticated API calls, `trigger` — cron triggers, `webhook` — webhook triggers, `mcp` — MCP tool invocations, `retry` — server-initiated retries, `rerun` — a user clicking **Re-run** on a past job, or `restart` — a user clicking **Restart from a step**; see [Re-running and Restarting Jobs](https://fremvaerk.github.io/stroem/guides/rerun-and-restart)). Child jobs from `type: task` actions do not trigger workspace hooks.
- If multiple YAML files define workspace-level hooks, they are merged (extended, not replaced).
- The same `hook.*` template context and `secret.*` variables are available as in task-level hooks.

## Cleanup and notification after a failure

A dependent's own `accept: [completed, failed]` (or `accept: any`) does let it run after an upstream failure, and that's fully decoupled from whether the job still fails (`continue_on_failure`, self-scoped on the upstream step — see [Conditionals](https://fremvaerk.github.io/stroem/guides/conditionals/)). But a step that isn't really part of the flow's own DAG — cleanup, notifying on-call, tearing down partial resources — still belongs in an `on_error` (or `on_cancel`) hook, so it fires exactly once on any terminal outcome without wiring an `accept` edge to every step that might fail:

```yaml
tasks:
  deploy:
    flow:
      deploy:
        action: deploy-app
    on_error:
      - action: run-cleanup
        input:
          env: "{{ hook.workspace }}"
```

## Task actions in hooks

Hook actions can be `type: task`, creating a full job for the referenced task instead of a single-step hook job. It is still a hook job (`source_type = "hook"`), not a child of the job that fired it — so the task keeps its own `retry:` and is not cancelled along with that job:

```yaml
tasks:
  deploy:
    flow:
      deploy:
        action: deploy-app
    on_error:
      - action: run-cleanup
        input:
          env: "{{ hook.workspace }}"
```

---

## Connections

Connections are named, typed configuration objects that store external system details (database credentials, API endpoints, etc.). They live in workspace YAML and are resolved automatically when used as task inputs.

## Connection Types

Define reusable schemas for connections using `connection_types`. Each property has a type, and can be marked as required, have a default value, or be flagged as secret.

```yaml
connection_types:
  postgres:
    host:
      type: string
      required: true
    port:
      type: integer
      default: 5432
    database:
      type: string
      required: true
    user:
      type: string
      required: true
    password:
      type: string
      required: true
      secret: true
```

Supported property types: `string`, `text`, `integer`, `number`, `boolean`, `date`, `datetime`.

## Connections

Define named connection instances under `connections`. The `type` field references a connection type; all other fields are values.

```yaml
connections:
  prod_db:
    type: postgres
    host: "db.example.com"
    port: 5432
    database: "myapp"
    user: "admin"
    password: "{{ 'ref+awsssm:///prod/db/password' | vals }}"

  staging_db:
    type: postgres
    host: "staging-db.internal"
    database: "myapp_staging"
    user: "app"
    password: "{{ 'ref+awsssm:///staging/db/password' | vals }}"
```

Connection values support Tera templates (e.g., `{{ 'ref+...' | vals }}` for secret resolution). Default values from the connection type are applied automatically for missing fields — in this example, `staging_db.port` will default to `5432`.

### Sharing a connection across workspaces

By default a connection is only usable within its own workspace. Set `shared: true` to allow other workspaces to reference it by qualified name (`workspace.connection-name`):

```yaml
connections:
  clickhouse-prod:
    type: clickhouse
    shared: true
    host: ch.internal
    password: "{{ secret.ch_password }}"
```

A reference to an unshared connection from another workspace is rejected with `400 Bad Request` at job creation for literal references; a templated flow-step value that resolves to an unshared connection is instead resolved at claim time and fails that step. See [Cross-Workspace References](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/#connections) for the full rules, including the claim-time case and how connection types are matched across workspaces.

### Untyped Connections

Connections without a `type` field are also valid. Untyped connections skip type validation entirely and can be used for any connection-type input field:

```yaml
connections:
  custom_api:
    url: "https://api.example.com"
    token: "abc123"
```

## Using Connections as Task Inputs

When a task input field's `type` matches a connection type name, the system treats it as a connection reference. At job creation time, the connection name string is automatically resolved to the full connection object.

```yaml
actions:
  run-migration:
    type: script
    script: "migrate --host {{ input.db.host }} --port {{ input.db.port }} --db {{ input.db.database }}"
    input:
      db:
        type: postgres        # Expects a resolved connection object

tasks:
  deploy:
    input:
      db:
        type: postgres        # References the connection type
      env:
        type: string           # Normal input
    flow:
      migrate:
        action: run-migration
        input:
          db: "{{ input.db }}"
```

When triggering the task, pass the connection name as the input value:

```json
{ "db": "prod_db", "env": "production" }
```

The system resolves `"prod_db"` to `{ "host": "db.example.com", "port": 5432, "database": "myapp", ... }`, making the full connection object available in templates as `{{ input.db.host }}`, `{{ input.db.port }}`, etc.

In the web UI, connection-type inputs render as a searchable dropdown listing all connections of that type in the workspace — no manual typing required.

When a connection type comes from a library, use the full namespaced name: `type: common.postgres` (where `common` is the library name).

### Cross-workspace

A connection-type input's `type:` and the connection name it resolves may each independently belong to another workspace (`type: jobs.clickhouse`, `default: jobs.clickhouse-prod`) — the type and the connection don't need to live in the same workspace. A foreign connection resolves only if its owner marked it `shared: true`. See [Cross-Workspace References → Connections](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/#connections) for the addressing rules, the `shared` gate, and redaction.

## Resolution Flow

**Task-level** (at job creation):
1. `merge_defaults()` runs as normal (fills missing fields, applies defaults)
2. `resolve_connection_inputs()` scans each task input field:
   - If the field type is a connection type (not `string`/`text`/`integer`/`number`/`boolean`), the provided string value is looked up in workspace connections
   - The string is replaced with the connection's values object
3. The resolved input is stored in the job — available in all templates

**Action-level** (at step execution):
Connection inputs on individual actions within a task are resolved when the step is about to execute, during the claim and render phase.

### Direct Object Pass-Through

Instead of passing a connection name string, you can pass a connection object directly:

```json
{ "db": { "host": "localhost", "port": 5432, "database": "test" } }
```

When an object is passed instead of a string, the resolution lookup is skipped and the object is used as-is.

## Validation

At YAML parse time (`stroem validate`):

- Connection type property types must be `string`, `text`, `integer`, `number`, or `boolean`
- Typed connections must reference an existing `connection_type`
- Required fields without defaults must be present in the connection
- Unknown fields produce warnings
- Empty string values produce errors
- Task input fields with non-primitive types must reference a known connection type
- A dotted (qualified) type reference, such as `type: jobs.clickhouse`, is not resolved offline — `stroem validate` emits a warning and defers the check to job creation, where the server can see every workspace

---

## Secrets & Encryption

Strøm supports two complementary approaches to secret management: **SOPS-encrypted workflow files** and **vals secret resolution** from external secret stores.

## SOPS encrypted files

Workflow files can be encrypted with [SOPS](https://github.com/getsops/sops). Name encrypted files with a `.sops.yaml` or `.sops.yml` suffix (e.g., `secrets.sops.yaml`). The server and CLI automatically detect these files and decrypt them before loading.

### Requirements

- `sops` must be installed and available on PATH
- Decryption keys must be configured (age, AWS KMS, GCP KMS, etc.)
- The `.sops.yaml` configuration file (if present) is ignored by the loader

### Example

```bash
# Encrypt a secrets file with age
sops -e --age age1... secrets.yaml > secrets.sops.yaml
```

The decrypted content is merged with other workflow files normally. Values from SOPS-encrypted files are already plaintext after decryption and don't need `| vals`.

## Secret resolution with `| vals`

The `| vals` filter resolves `ref+` secret references at template render time using the [vals](https://github.com/helmfile/vals) CLI.

### Prerequisites

The `vals` binary must be installed and available on PATH on the **server** (where templates are rendered).

### Workspace-level secrets

The `secrets:` section is rendered through Tera at **workspace load time**. Secrets are resolved once and cached in memory:

```yaml
secrets:
  DB_PASSWORD: "{{ 'ref+awsssm:///prod/db/password' | vals }}"
  API_TOKEN: "{{ 'ref+vault://secret/data/api#token' | vals }}"
  SLACK_WEBHOOK: "{{ 'ref+gcpsecrets://my-project/slack-webhook' | vals }}"
```

Then use the resolved values in templates — no `| vals` needed:

```yaml
env:
  DB_PASSWORD: "{{ secret.DB_PASSWORD }}"
script: "deploy --token {{ secret.API_TOKEN }}"
```

Secrets are re-resolved when the workspace reloads (on config change or git poll), so rotated secrets are picked up automatically.

### Inline resolution

You can also use `| vals` inline in any template expression (step `input:`, action `env:`, `script:`, `source:`, hook `input:`) without going through `secrets:`:

```yaml
env:
  DB_PASSWORD: "{{ 'ref+awsssm:///prod/db/password' | vals }}"
```

This resolves at step claim time rather than workspace load time.

### Full example

```yaml
secrets:
  DB_PASSWORD: "{{ 'ref+awsssm:///prod/db/password' | vals }}"
  API_TOKEN: "{{ 'ref+vault://secret/data/api#token' | vals }}"
  SLACK_WEBHOOK: "{{ 'ref+gcpsecrets://my-project/slack-webhook' | vals }}"

actions:
  deploy:
    type: script
    script: "deploy --token {{ secret.API_TOKEN }}"
    env:
      DB_PASSWORD: "{{ secret.DB_PASSWORD }}"
    input:
      env: { type: string }

tasks:
  deploy:
    flow:
      deploy:
        action: deploy
        input:
          env: "{{ input.env }}"
    on_success:
      - action: notify-slack
        input:
          webhook_url: "{{ secret.SLACK_WEBHOOK }}"
          message: "Deploy succeeded"
```

### Supported backends

Any backend supported by vals works:

- `ref+awsssm://` — AWS SSM Parameter Store
- `ref+vault://` — HashiCorp Vault
- `ref+gcpsecrets://` — Google Cloud Secret Manager
- `ref+azurekeyvault://` — Azure Key Vault
- `ref+sops://` — SOPS encrypted files
- And [many more](https://github.com/helmfile/vals#supported-backends)

### Behavior

- Plain strings (not starting with `ref+`) pass through unchanged — `{{ "hello" | vals }}` returns `"hello"`
- Non-string values (numbers, booleans) pass through unchanged
- If `vals` is not installed and a `ref+` value is encountered, the template render fails with a clear error
- Each `| vals` usage invokes the vals CLI once

### What is masked

Job detail, webhook and MCP responses mask secret values that appear in **strings** — including strings inside JSON input values. Numbers, booleans and `null` are never masked:

- A boolean or `null` cannot be hidden by masking: it carries one bit, and masking every `true`, `false` or `null` in a response would destroy it.
- Quote numeric secrets (`PORT: "5432"`) so they are masked; `stroem validate` warns about unquoted ones.
- A filter that changes a secret's form — `| int`, `| upper`, `| b64encode`, a slice — produces a value that is not masked, in any field type.
- Values of 3 characters or fewer are never masked.

### API redaction

Secret values are automatically redacted from API responses. When you view a job via `GET /api/jobs/:id`, every field except identifiers (ids, names, statuses, revisions, timestamps) — job input/output, step input/output, approval messages, conditions — that contains a known secret value will have it replaced with `••••••`. Substring matches are also redacted. Additionally, unresolved `ref+` references are redacted to avoid leaking secret-manager paths.

The same masking applies to a sync webhook's response and the webhook
job-status poll, to MCP `get_job_status`, and to the step errors listed on a
worker's detail page. A job that runs on a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) is
also masked with the secrets of its commit (and of every commit it is
connected to); see [Git Refs → Secrets and redaction](https://fremvaerk.github.io/stroem/guides/git-refs/#secrets-and-redaction).
Job **logs** are not masked: a script that prints a secret shows it in the
log.

Step `error_message` and `retry_history` are redacted the same way. Template
errors never contain values: they are built from a fixed category, the
position in the template and at most a type name or the failing filter's name
— not Tera's own message text, not the rendered value, not the template line.
Errors about a value *after* rendering (for example a rendered connection name
that does not resolve) name the input field, never the value. As a second
layer, errors raised while rendering a step at claim time are also scrubbed
against the known secret values **before** they are written. To see Tera's full
report, run the workflow locally with `stroem run` or `stroem validate`. Limits: `stroem validate` compiles only `when`, `for_each` and agent prompts (plus secrets and connections at load), and `stroem run` only runs tasks made entirely of local `type: script` steps.

**CAUTION:** `{{ obj }}` renders a whole object. `"dsn={{ input.db }}"` with a
connection-typed input writes every property of the connection — password
included — into the step input (Tera 1 wrote `[object]`). The value is masked
when the job is read if the property is `secret: true`, but it is still the
input the step receives. Reference the fields you need (`{{ input.db.host }}`).

---

## Agent Actions

Agent actions let you call LLMs as first-class workflow steps. Supports 19 LLM providers including Anthropic, OpenAI, Gemini, Groq, and more. Generate text, classify content, extract data, or feed LLM responses directly into downstream steps.

## Quick Start

### 1. Configure worker for agent execution

Workers need to be configured with LLM provider API keys. In your `worker-config.yaml`:

```yaml
agents:
  providers:
    - id: anthropic-main
      type: anthropic
      api_key: "${ANTHROPIC_API_KEY}"
      model: claude-opus-4-1-20250805
      max_tokens: 2048
      temperature: 0.7
      max_retries: 2

    - id: openai-gpt4
      type: openai
      api_key: "${OPENAI_API_KEY}"
      model: gpt-4o
      max_tokens: 1024

# Worker must also declare the "agent" capability
capabilities: ["script", "agent"]
```

**Server configuration** (optional, for validation only):

Optionally define providers in `server-config.yaml` for YAML validation at load time. The server does NOT store API keys.

```yaml
agents:
  providers:
    - id: anthropic-main
      type: anthropic
      model: claude-opus-4-1-20250805
    - id: openai-gpt4
      type: openai
      model: gpt-4o
```

### 2. Define an agent action

In your workflow YAML:

```yaml
actions:
  classify-ticket:
    type: agent
    provider: anthropic-main
    system_prompt: |
      You are a support ticket classifier.
      Classify tickets as bug, feature_request, or question.
    prompt: |
      Classify this ticket:
      Subject: {{ input.subject }}
      Body: {{ input.body }}
    output:
      category:
        type: string
        required: true
        options: [bug, feature_request, question]
      confidence:
        type: number
        required: true
      summary:
        type: string
    input:
      subject: { type: string, required: true }
      body: { type: string, required: true }
```

### 3. Use in a flow

```yaml
tasks:
  support-workflow:
    input:
      ticket_subject: { type: string }
      ticket_body: { type: string }
    flow:
      classify:
        action: classify-ticket
        input:
          subject: "{{ input.ticket_subject }}"
          body: "{{ input.ticket_body }}"

      route-bug:
        action: escalate-to-engineering
        depends_on: [classify]
        when: "{{ classify.output.category == 'bug' }}"
        input:
          ticket_id: "{{ input.ticket_id }}"

      route-feature:
        action: add-to-roadmap
        depends_on: [classify]
        when: "{{ classify.output.category == 'feature_request' }}"

      route-question:
        action: send-faq-response
        depends_on: [classify]
        when: "{{ classify.output.category == 'question' }}"
```

## Provider Configuration

Agent actions execute on workers, which load providers from their `worker-config.yaml`. The server optionally validates provider names from `server-config.yaml`.

### Supported Providers

The following 19 providers are supported:

| Provider | Type | Description | Requires API Key |
|----------|------|-------------|------------------|
| Anthropic | `anthropic` | Claude models | Yes |
| Azure | `azure` | Azure OpenAI Service | Yes (requires `api_endpoint`) |
| Cohere | `cohere` | Cohere models | Yes |
| DeepSeek | `deepseek` | DeepSeek models | Yes |
| Galadriel | `galadriel` | Galadriel models (OpenAI-compatible API) | Yes |
| Gemini | `gemini` | Google Gemini | Yes |
| Groq | `groq` | Groq models | Yes |
| Hugging Face | `huggingface` | Hugging Face models | Yes |
| Hyperbolic | `hyperbolic` | Hyperbolic AI | Yes |
| Llamafile | `llamafile` | Local Llamafile or llama.cpp `llama-server` (`api_endpoint` is the server root, without `/v1`) | No |
| Mira | `mira` | Mira AI | Yes |
| Mistral | `mistral` | Mistral models | Yes |
| Moonshot | `moonshot` | Moonshot AI | Yes |
| Ollama | `ollama` | Local Ollama server | No |
| OpenAI | `openai` | GPT models | Yes |
| OpenRouter | `openrouter` | OpenRouter API | Yes |
| Perplexity | `perplexity` | Perplexity AI | Yes |
| Together | `together` | Together AI | Yes |
| xAI | `xai` | xAI Grok | Yes |

### Provider Examples

**Anthropic:**

```yaml
agents:
  providers:
    - id: anthropic-main
      type: anthropic
      api_key: "${ANTHROPIC_API_KEY}"
      model: claude-opus-4-1-20250805
      max_tokens: 2048
      temperature: 0.7
      max_retries: 2
```

**Gemini:**

```yaml
agents:
  providers:
    - id: gemini-pro
      type: gemini
      api_key: "${GEMINI_API_KEY}"
      model: gemini-2.0-flash
      max_tokens: 1024
```

**Groq:**

```yaml
agents:
  providers:
    - id: groq-fast
      type: groq
      api_key: "${GROQ_API_KEY}"
      model: llama-3.3-70b-versatile
      max_tokens: 2048
```

**Ollama (local, no API key required):**

```yaml
agents:
  providers:
    - id: ollama-local
      type: ollama
      api_endpoint: "http://localhost:11434"
      model: llama2
```

**Azure OpenAI:**

```yaml
agents:
  providers:
    - id: azure-gpt4
      type: azure
      api_key: "${AZURE_API_KEY}"
      api_endpoint: "https://myresource.openai.azure.com"
      model: gpt-4-deployment-name
      max_tokens: 2048
```

**OpenAI-compatible endpoint:**

```yaml
agents:
  providers:
    - id: vllm-local
      type: openai
      api_key: "${CUSTOM_API_KEY}"
      api_endpoint: "http://localhost:8000/v1"
      model: local-model
```

### Configuration Fields

| Field | Required | Description |
|-------|----------|-------------|
| `id` | Yes | Unique provider identifier used in actions |
| `type` | Yes | Provider type (see table above) |
| `api_key` | Conditional | API key for the provider. Not required for `ollama` and `llamafile`. Supports env var templating with `${VAR_NAME}` |
| `api_endpoint` | Conditional | Custom endpoint URL. Required for `azure`; optional for OpenAI-compatible servers |
| `model` | Yes | Model identifier (e.g., `claude-opus-4-1-20250805`, `gpt-4o`, `gemini-2.0-flash`) |
| `max_tokens` | No | Default max completion tokens (can be overridden per action) |
| `temperature` | No | Default sampling temperature (0–2) |
| `max_retries` | No | Number of retries on transient errors (default 2) |

## Action Fields

| Field | Required | Type | Description |
|-------|----------|------|-------------|
| `type` | Yes | String | Must be `agent` |
| `provider` | Yes | String | Provider ID from config |
| `prompt` | Yes | String | Tera template for the user message |
| `system_prompt` | No | String | Tera template for system/instruction message |
| `output` | No | OutputDef | Structured output schema (converted to JSON Schema at dispatch time) |
| `model` | No | String | Override provider's default model |
| `max_tokens` | No | Integer | Override provider's max tokens |
| `temperature` | No | Number | Override provider's temperature |
| `input` | No | Object | Input schema (same as other actions) |
| `timeout` | No | Duration | Max time for LLM call (default 5m) |

## Structured Output

When `output` is set, it is converted to JSON Schema at dispatch time and the LLM is instructed to respond with JSON matching the schema. If parsing fails, the step fails with an error.

### Output field properties

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | String | Required | `string`, `integer`, `number`, `boolean`, `array`, `object` |
| `description` | String | — | Human-readable description (included in JSON Schema) |
| `required` | Boolean | `false` | Whether the field must be present in the response |
| `default` | Any | — | Default value (included in JSON Schema) |
| `options` | Array | — | Allowed values (maps to JSON Schema `enum`) |

### With output

```yaml
actions:
  analyze:
    type: agent
    provider: anthropic-main
    prompt: "Analyze this data: {{ input.data }}"
    output:
      sentiment:
        type: string
        required: true
        options: [positive, negative, neutral]
      keywords:
        type: array
      score:
        type: number
```

**Output:**

```json
{
  "sentiment": "positive",
  "keywords": ["great", "amazing"],
  "score": 0.95,
  "_meta": {
    "provider": "anthropic",
    "model": "claude-opus-4-1-20250805",
    "tokens": {
      "input": 156,
      "output": 42
    },
    "latency_ms": 1234
  }
}
```

### Without output

```yaml
actions:
  summarize:
    type: agent
    provider: anthropic-main
    prompt: "Summarize: {{ input.text }}"
    # No output — response captured as text
```

**Output:**

```json
{
  "text": "This is a summary of the provided text...",
  "_meta": {
    "provider": "anthropic",
    "model": "claude-opus-4-1-20250805",
    "tokens": {
      "input": 128,
      "output": 87
    },
    "latency_ms": 987
  }
}
```

> **Note:** The `_meta` key is reserved. If your `output` defines a `_meta` property, it will be overwritten by the system metadata.

### Using _meta in downstream steps

```yaml
tasks:
  workflow:
    flow:
      analyze:
        action: analyze

      log-usage:
        action: log-api-usage
        depends_on: [analyze]
        input:
          tokens: "{{ analyze.output._meta.tokens.input + analyze.output._meta.tokens.output }}"
          latency_ms: "{{ analyze.output._meta.latency_ms }}"
```

## Prompt Templating

Both `prompt` and `system_prompt` are Tera templates. They're rendered at step execution time with access to:

| Variable | Description |
|----------|-------------|
| `input.*` | Job-level input |
| `<step_name>.output.*` | Output from completed upstream steps |
| `secret.*` | Workspace secrets |
| `job.revision` | Workspace revision pinned on the job at creation |
| `each.item` / `each.index` / `each.total` | Loop variables, when the agent step is an instance of a [`for_each` loop](https://fremvaerk.github.io/stroem/guides/loops/) |

:::caution[A template that cannot be rendered fails the step]
If `prompt` or `system_prompt` references a variable that does not resolve, the
step fails at claim time with a value-free template error (for example
`undefined variable or field (line 1, column 4)`), and that message is recorded on the step.

Earlier versions swallowed the render error: a failing `prompt` surfaced as the
misleading `Agent step has no rendered prompt`, and a failing `system_prompt`
was ignored entirely — the agent ran **without** its system prompt. Both now
report the real cause.
:::

**Example with templating:**

```yaml
actions:
  generate-report:
    type: agent
    provider: anthropic-main
    system_prompt: |
      You are a report generator for the {{ input.department }} department.
      Use a formal, professional tone.
    prompt: |
      Generate a report for Q{{ input.quarter }} {{ input.year }}.
      Previous metrics: {{ previous-step.output.metrics }}
      API key for data service: {{ secret.data_api_key }}
    input:
      department: { type: string }
      quarter: { type: integer }
      year: { type: integer }
```

**Step name rules**: Step names with hyphens become underscores in templates. A step named `previous-step` is referenced as `previous_step.output.*`.

## Error Handling

Agent actions can fail for several reasons. Each produces a specific error message:

| Error | Cause | Resolution |
|-------|-------|-----------|
| `Provider '{id}' not found` | `provider` ID doesn't exist in config | Check provider ID in action and config |
| `Template rendering error: ...` | `prompt` or `system_prompt` syntax invalid | Fix Tera template syntax or variable references |
| `LLM API error: ...` | Network error, API quota exceeded, invalid key | Check API credentials, rate limits, network connectivity |
| `JSON parsing failed: ...` | Response doesn't match `output` | Adjust schema or prompt to guide LLM response format |
| `Step timeout exceeded` | LLM call took longer than `timeout` | Increase timeout or reduce `max_tokens` |

**Example: Handling LLM failures with conditions**

```yaml
tasks:
  robust-workflow:
    flow:
      classify:
        action: classify-ticket

      handle-failure:
        action: escalate-to-human
        depends_on: [classify]
        when: "{{ classify.status == 'failed' }}"
        # Runs only if classify step failed

      process-result:
        action: process-classification
        depends_on: [classify]
        when: "{{ classify.status == 'completed' }}"
        # Runs only if classify succeeded
```

## Limitations (Phase 7A)

- **Single-turn only**: Agents cannot have multi-turn conversations. Each step is an independent LLM call. Multi-turn support with tools is planned for Phase 7B.
- **No streaming**: Responses are fully buffered before being captured. Streaming output is not yet supported.
- **No request/response logging**: Full LLM conversation history is not logged (only metadata like token count and latency).
- **No `ref:` with agents**: a flow step may not name an agent action with [`ref:`](https://fremvaerk.github.io/stroem/guides/git-refs/) (`400` at job creation), and a `tools: [{task: …}]` entry may not carry `ref` (a validation error; a call naming it is rejected). Agent steps *inside* a job that runs on a git ref use that commit's definitions, and their task-tool child jobs run at the same commit.

## Common Patterns

### Classification with routing

Classify input and route to different steps based on the result:

```yaml
actions:
  classify-issue:
    type: agent
    provider: anthropic-main
    prompt: "Classify this GitHub issue: {{ input.issue_body }}"
    output:
      category:
        type: string
        required: true
        options: [bug, feature, documentation, wontfix]

tasks:
  issue-workflow:
    input:
      issue_body: { type: string }
    flow:
      classify:
        action: classify-issue
        input:
          issue_body: "{{ input.issue_body }}"

      create-bug-ticket:
        action: create-jira-ticket
        depends_on: [classify]
        when: "{{ classify.output.category == 'bug' }}"

      add-feature-label:
        action: add-github-label
        depends_on: [classify]
        when: "{{ classify.output.category == 'feature' }}"
```

### Data extraction

Extract structured data from unstructured text:

```yaml
actions:
  extract-contact:
    type: agent
    provider: anthropic-main
    prompt: |
      Extract contact information from this text:
      {{ input.text }}

      Return as JSON with name, email, phone.
    output:
      name: { type: string }
      email: { type: string }
      phone: { type: string }

tasks:
  contact-extraction:
    flow:
      extract:
        action: extract-contact
        input:
          text: "{{ input.unstructured_data }}"

      save-contact:
        action: save-to-db
        depends_on: [extract]
        input:
          name: "{{ extract.output.name }}"
          email: "{{ extract.output.email }}"
```

### Conditional processing based on analysis

Analyze content and conditionally trigger different workflows:

```yaml
actions:
  analyze-sentiment:
    type: agent
    provider: anthropic-main
    prompt: "Analyze sentiment of: {{ input.text }}"
    output:
      sentiment:
        type: string
        required: true
        options: [positive, negative, neutral]
      score: { type: number }

tasks:
  feedback-workflow:
    input:
      feedback: { type: string }
    flow:
      analyze:
        action: analyze-sentiment
        input:
          text: "{{ input.feedback }}"

      positive-branch:
        action: send-thank-you
        depends_on: [analyze]
        when: "{{ analyze.output.sentiment == 'positive' }}"

      negative-branch:
        action: create-support-ticket
        depends_on: [analyze]
        when: "{{ analyze.output.sentiment == 'negative' }}"
```

## Security Considerations

### Secrets in prompts

Workspace secrets are available in prompt templates via `{{ secret.NAME }}`. When an agent step renders a secret into a prompt, that secret is sent to the external LLM API (Anthropic, OpenAI, or whatever is configured). This is the same access model as script steps (which can read secrets via environment variables), but with agent steps the data leaves your infrastructure.

**Recommendations:**
- Only reference secrets in agent prompts when necessary
- Use the most restrictive LLM provider data policies available
- Consider using separate, limited-scope API keys as secrets for agent-accessible workflows
- Review workflow YAML for unintended secret references before deploying

### Custom API endpoints

The `api_endpoint` field in provider configuration accepts arbitrary URLs. Administrators should ensure endpoints point only to trusted LLM-compatible API servers. The field is only configurable via `server-config.yaml` (not by workflow authors).

## Cost Optimization

Agent steps incur LLM API costs. Keep these strategies in mind:

- **Reduce max_tokens**: Lower `max_tokens` for shorter responses (e.g., classification often needs < 100 tokens)
- **Reuse for multiple steps**: Design workflows so one agent step can answer multiple questions, if possible
- **Cache results**: For repeated analyses, consider caching outputs in a database step and checking before calling the agent
- **Batch operations**: Where possible, batch multiple items into one prompt instead of separate calls
- **Model selection**: Use faster/cheaper models for simple tasks; reserve advanced models for complex analysis

**Example cost-optimized classification:**

```yaml
actions:
  quick-classify:
    type: agent
    provider: anthropic-main
    model: claude-haiku-3-5-20241022  # Faster, cheaper model
    max_tokens: 50  # Minimal output
    prompt: |
      Classify as: [bug, feature, wontfix]
      {{ input.title }}
```

---

## Task State Snapshots

Tasks in Strøm can persist both files and structured data between runs using state snapshots. This enables workflows like SSL certificate renewal, incremental processing with cursors, and caching build artifacts.

## Overview

State snapshots are immutable archives created after a step completes, containing:

- **Files**: Any artifacts written to `/state-out` (certificates, keys, compiled binaries, caches)
- **Structured data**: JSON objects emitted via the `STATE:` protocol (counters, cursors, metadata, timestamps)

Each task maintains a rolling history of state snapshots. When a new job starts, all steps have access to the latest snapshot from the previous run.

## How it works

### Writing state

State is written in two ways:

#### File-based state

Write files to the `/state-out` directory (or `$STATE_OUT_DIR` environment variable):

```bash
#!/bin/bash
# Renew SSL certificate
openssl req -new -x509 -days 365 -nodes \
  -out /etc/letsencrypt/live/example.com/fullchain.pem \
  -keyout /etc/letsencrypt/live/example.com/privkey.pem

# Persist the certificate for the next job
cp /etc/letsencrypt/live/example.com/fullchain.pem $STATE_OUT_DIR/cert.pem
cp /etc/letsencrypt/live/example.com/privkey.pem $STATE_OUT_DIR/privkey.pem

echo "Certificate renewed"
```

Files are automatically collected into a gzip tarball and stored after the step completes.

#### Structured state (STATE: protocol)

Emit JSON to stdout using the `STATE:` prefix to store structured data:

```bash
#!/bin/bash
# Check certificate expiry and emit state
DAYS=$(openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem \
  -noout -enddate | cut -d= -f2)

echo "STATE: {\"expiry_date\": \"$DAYS\", \"checked_at\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
echo "STATE: {\"domain\": \"example.com\"}"  # Multiple lines are deep-merged
```

Multiple `STATE:` lines in the same step are automatically merged (last write wins for duplicate keys):

```bash
echo "STATE: {\"cursor\": \"page-1\", \"total_pages\": 50}"
echo "STATE: {\"cursor\": \"page-2\"}"  # Overwrites cursor, keeps total_pages
# Result: {"cursor": "page-2", "total_pages": 50}
```

Both approaches work together — files and structured state are stored in the same snapshot. File state becomes `state.json` in the tarball.

### Reading state

#### Previous files

Access the previous state directory via the `/state` mount (or `$STATE_DIR` environment variable):

```bash
#!/bin/bash
# Check if a certificate exists from a previous run
if [ -f "$STATE_DIR/cert.pem" ]; then
    echo "Found previous certificate, checking expiry..."
    openssl x509 -in "$STATE_DIR/cert.pem" -noout -enddate
else
    echo "No previous certificate found"
fi
```

The `/state` directory is read-only and contains the contents of the latest snapshot.

#### In templates

Structured state is available in Tera templates as the `state` object. Use it to conditionally execute steps:

```yaml
tasks:
  renew-cert:
    flow:
      check-expiry:
        action: check-ssl-expiry
        input:
          domain: "{{ input.domain }}"

      renew-cert:
        action: renew-ssl
        depends_on: [check-expiry]
        # Only run if no previous state, or less than 30 days remain
        when: "{{ not state or state.days_remaining < 30 }}"
        input:
          domain: "{{ input.domain }}"

      upload-cert:
        action: upload-to-cdn
        depends_on: [renew-cert]
        input:
          domain: "{{ state.domain | default(value='example.com') }}"
```

If no previous state exists, `state` is `null` in the template context — use `not state` to guard operations, or `state.key | default(value=…)`, which falls back to the default on a first run.

## Runner behavior

| Runner | `/state` (read) | `/state-out` (write) | `STATE:` protocol |
|--------|-----------------|---------------------|-------------------|
| Shell (local) | `$STATE_DIR` path | `$STATE_OUT_DIR` path | Yes |
| Docker | Bind mount `:ro` at `/state` | Bind mount `:rw` at `/state-out` | Yes |
| Kubernetes | emptyDir volume | emptyDir volume | Yes |

All runners set `STATE_DIR` and `STATE_OUT_DIR` environment variables automatically.

**NOTE:** For Kubernetes runners, file-based state from previous runs is available via `/state`. However, uploading file state (`/state-out`) in the initial release works only for the `STATE:` protocol (structured data). File artifacts are supported on shell and Docker runners.
## Multi-step tasks

When a task has multiple steps:

- **All steps** start with the same previous state snapshot
- **Sequential steps** see state written by earlier steps (state is resolved at claim time)
- **Parallel steps** all see the pre-parallel state independently; each can write new state
- Only steps that write to `/state-out` or emit `STATE:` lines create a new snapshot

**Example: Sequential state propagation**

```yaml
tasks:
  incremental-process:
    flow:
      load-cursor:
        action: fetch-cursor        # Reads $STATE_DIR/cursor.json, emits STATE:

      process-batch:
        action: process-data        # Reads $STATE_DIR/cursor.json (updated by step 1)
        depends_on: [load-cursor]   # Sees step 1's state

      save-cursor:
        action: save-cursor         # Writes new cursor to $STATE_OUT_DIR
        depends_on: [process-batch]
```

In this flow:
1. `load-cursor` runs first, reads previous cursor from `/state`, emits new cursor via `STATE:`
2. `process-batch` is claimed next and sees the updated state (step 1's `STATE:` JSON)
3. `save-cursor` also sees the updated state

## Configuration

State storage uses the same archive backend as [log storage](https://fremvaerk.github.io/stroem/operations/log-storage/) by default. Override with a dedicated section:

```yaml
# server-config.yaml
state_storage:
  prefix: "state/"          # archive key prefix (default: "state/")
  max_snapshots: 5          # snapshots retained per task (default: 5)
  # Optional: override the archive backend
  # archive:
  #   type: s3
  #   bucket: my-state-bucket
  #   region: eu-west-1
  #   endpoint: "https://s3.example.com"  # optional (for S3-compatible services)
```

If neither `state_storage` nor `log_storage.archive` is configured, the worker cannot download or upload snapshots — state directories remain empty, no new snapshots are stored, and the claim response carries no snapshot keys. Templates still resolve `{{ state.* }}` / `{{ global_state.* }}` from snapshot rows already recorded: migration 047 persists the parsed sidecar on the row itself, so rendering reads it straight from the database rather than the archive.

### Archive key format

State snapshots are stored with this key structure:

```
{prefix}{workspace}/{task_name}/{job_id}.tar.gz
```

Example: `state/production/renew-ssl/550e8400-e29b-41d4-a716-446655440000.tar.gz`

## Environment variables

| Variable | Value | Availability |
|----------|-------|--------------|
| `STATE_DIR` | Path to the extracted previous state (read-only) | All runners |
| `STATE_OUT_DIR` | Path for writing new state | All runners |

These are set automatically by the worker and runner. If no previous state exists, `STATE_DIR` points to an empty directory.

## Retention

Snapshots are pruned automatically per task to keep a rolling window of recent snapshots:

- Default: keep the last **5 snapshots** per task (configurable via `max_snapshots`) — per task **and** per ref, see [Pinned jobs](#pinned-jobs)
- Older snapshots are deleted from both the database and archive
- Snapshots survive job retention cleanup — the snapshot's `job_id` reference is cleared when the job is deleted, but the snapshot remains in the archive

## Which state a step sees

State belongs to the **job**: every step of a job reads and writes the state
of the job's own workspace and task. That includes a step that runs a
[cross-workspace action](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/): it reads and
writes the calling job's state, not the action owner's. The server takes
these coordinates from the job, never from the worker's request.

### Pinned jobs

A job running on a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) reads and writes state
partitioned by the ref string: `release/2.3`'s snapshots are separate from
`release/2.4`'s and from the default branch's. Bumping a ref (`v2.3.1` →
`v2.3.2`) starts with empty state. The same holds for global state: a pinned
job's `GLOBAL_STATE:` and `{{ global_state.* }}` use that workspace's
partition for the ref.

Workers older than the server do not tell the server which job a state
download is for, so a pinned job claimed by such a worker gets the default
(unpinned) partition mounted at `/state`. Its templates (`{{ state.* }}`)
and its uploads still use the right partition, because the server renders
templates and decides where uploads go. Upgrade workers together with the
server.

## Use cases

### SSL certificate renewal

```yaml
actions:
  check-ssl:
    type: script
    script: |
      #!/bin/bash
      if [ -f "$STATE_DIR/cert.pem" ]; then
        EXPIRY=$(openssl x509 -in "$STATE_DIR/cert.pem" -noout -enddate | cut -d= -f2)
        DAYS=$(( ($(date -d "$EXPIRY" +%s) - $(date +%s)) / 86400 ))
        echo "STATE: {\"days_remaining\": $DAYS, \"expiry\": \"$EXPIRY\"}"
      else
        echo "STATE: {\"days_remaining\": 0, \"expiry\": null}"
      fi

  renew-ssl:
    type: script
    script: |
      #!/bin/bash
      # Use certbot or acme.sh to renew
      certbot renew
      cp /etc/letsencrypt/live/example.com/fullchain.pem $STATE_OUT_DIR/cert.pem
      cp /etc/letsencrypt/live/example.com/privkey.pem $STATE_OUT_DIR/privkey.pem

tasks:
  maintain-cert:
    flow:
      check:
        action: check-ssl

      renew:
        action: renew-ssl
        depends_on: [check]
        when: "{{ not state or state.days_remaining < 30 }}"
```

### Incremental processing with cursors

```yaml
actions:
  process-logs:
    type: script
    script: |
      #!/bin/bash
      # Load cursor from previous state
      CURSOR=$(cat "$STATE_DIR/cursor.json" 2>/dev/null | jq -r .offset // "0")

      # Process logs starting from cursor
      tail -n +$CURSOR /var/log/app.log | while read line; do
        echo "$line" | process-line
      done

      # Save new cursor position
      NEW_CURSOR=$(wc -l < /var/log/app.log)
      echo "STATE: {\"offset\": $NEW_CURSOR, \"last_processed\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"

tasks:
  log-processor:
    flow:
      ingest:
        action: process-logs
```

### Build artifact caching

```yaml
actions:
  build:
    type: script
    runner: docker
    language: javascript
    dependencies: [next]
    script: |
      cd /workspace
      # Restore cache from previous build
      if [ -d "$STATE_DIR/.next/cache" ]; then
        cp -r "$STATE_DIR/.next/cache" .next/
      fi

      # Build and cache artifacts
      npm run build
      mkdir -p $STATE_OUT_DIR/.next
      cp -r .next/cache $STATE_OUT_DIR/.next/
      echo "STATE: {\"build_time\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\", \"size_mb\": $(du -sm .next | cut -f1)}"

tasks:
  web-build:
    flow:
      build:
        action: build
```

## Global workspace state

In addition to per-task state, Strøm supports **global workspace state** — shared across all tasks in a workspace. Any task can read and write it.

### Writing global state

Use the `GLOBAL_STATE:` protocol or write files to `/global-state-out` (`$GLOBAL_STATE_OUT_DIR`):

```bash
#!/bin/bash
# Share a token across all tasks in this workspace
echo "GLOBAL_STATE: {\"api_token\": \"$(vault read -field=token secret/api)\", \"refreshed_at\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
```

### Reading global state

Previous global state is available at `/global-state` (`$GLOBAL_STATE_DIR`) and in Tera templates as `{{ global_state.* }}`:

```yaml
tasks:
  deploy:
    flow:
      deploy:
        action: deploy-app
        input:
          token: "{{ global_state.api_token }}"
```

### How it differs from task state

| | Task state | Global state |
|---|---|---|
| Scope | Per task | Per workspace (all tasks) |
| Protocol | `STATE:` | `GLOBAL_STATE:` |
| Read dir | `/state` | `/global-state` |
| Write dir | `/state-out` | `/global-state-out` |
| Templates | `{{ state.* }}` | `{{ global_state.* }}` |
| Concurrent writes | Per-task (rare) | Cross-task (last writer wins) |

Both types of state are available simultaneously — a step can read and write both task state and global state in the same run.

## Uploading state manually

State snapshots are normally written by tasks, but operators can also upload a snapshot out-of-band via the API. This is useful for bootstrapping — for example, seeding a Let's Encrypt certificate so a renewal task has something to read on first run.

### Endpoints

```
POST /api/workspaces/{ws}/tasks/{task}/state    # task-state
POST /api/workspaces/{ws}/state                 # global workspace state (admin)
```

Both accept a gzip tarball (max 50 MB) in the request body and state-JSON key/value pairs as query parameters.

Manual uploads always write the default (unpinned) partition; there is no
way yet to upload state for a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/)'s partition.

### Auth

- Task-state: requires `Run` permission on the task (same permission as triggering it via `/execute`).
- Global-state: requires admin.

### Query parameters

| Parameter | Default | Meaning |
|---|---|---|
| `mode` | `replace` | `replace` = snapshot contains exactly what you upload; `merge` = overlay on top of the latest snapshot |
| any other `key=value` | — | becomes a top-level string entry in `state.json` |

### Modes

- **Replace** (default): the new snapshot's files are exactly the tarball's contents plus `state.json` synthesised from query params. Any file or key present in the previous snapshot and absent from the upload is dropped.
- **Merge**: the new snapshot starts from the previous snapshot's contents, then uploaded files overlay same-named files (new wins), and query-param state keys shallow-merge over the previous `state.json` (new wins).

Merge cannot delete files or state keys — to remove something, use `mode=replace` and omit it.

### The tarball

- Include any files you want available at `$STATE_DIR` on the next run.
- **Do not** include a root-level `state.json` — pass state values via query parameters instead. Nested paths like `subdir/state.json` are allowed.
- An empty tarball is valid; an empty body (Content-Length 0) is also valid and is treated as an empty tarball.

### Examples

Bootstrap a Let's Encrypt certificate for a renewal task:

```bash
stroem-api state upload \
    --workspace production \
    --task renew-ssl \
    --state domain=example.com \
    --state expiry=2026-07-21 \
    fullchain.pem privkey.pem
```

Rotate only the expiry value (merge mode, no file changes):

```bash
stroem-api state upload \
    --workspace production \
    --task renew-ssl \
    --mode merge \
    --state expiry=2027-07-21
```

Raw curl:

```bash
tar -czf /tmp/certs.tar.gz fullchain.pem privkey.pem
curl -X POST \
    -H "Authorization: Bearer $STROEM_TOKEN" \
    -H "Content-Type: application/gzip" \
    --data-binary @/tmp/certs.tar.gz \
    "$STROEM_URL/api/workspaces/production/tasks/renew-ssl/state?domain=example.com&expiry=2026-07-21"
```

### Audit

Each upload creates a synthetic job row with `source_type="upload"` and `status="completed"`. Filter the jobs list with `?source_type=upload` to see upload history. The job's `input` JSON records the uploader, size, and SHA-256 of the stored tarball.

### Caveats

- Values are **strings** — use `| int` / `| float` in templates if type coercion matters. `| int` and `| float` raise an error on input that does not parse as a number (they no longer return `0`). There is no `bool` filter; compare with `== 'true'` instead.
- Total query-string size is limited by the web server (~8 KB typical). For larger state, emit via the `STATE:` protocol from a task run.
- Do **not** put secrets in query parameters — they land in access logs and tracing spans. Use the secret backend (see [Secrets & Encryption](https://fremvaerk.github.io/stroem/guides/secrets/)) instead.
- To remove files or state keys, use `mode=replace`.

## Limitations

- State is immutable once a snapshot is created. To update state, emit new `STATE:`/`GLOBAL_STATE:` lines or write new files from the step.
- The maximum size of a state snapshot is **50 MB**. Oversized uploads are rejected with a warning (non-fatal — the job continues).
- For Kubernetes runners, file-based state (`/state-out`) works via the `STATE:` protocol. Direct file uploads are supported on shell and Docker runners.
- Global state concurrent writes don't merge — last writer wins. Coordinate via task dependencies if ordering matters.

---

## Git Refs

A reference to an action or a task can carry a `ref:` — a **branch**, a
**tag** or a full **commit SHA** of the git workspace that owns it. The
target then runs from that commit: its YAML (flow, inputs, action
definitions, hooks, secrets, connections) and its files. The typical use is
several releases of one workspace running side by side, with the default
branch as the manifest that pins which release each step, task or trigger
runs.

```yaml
# on main
tasks:
  daily:
    flow:
      import:
        action: import            # own workspace @ release/2.3
        ref: release/2.3
      export:
        action: billing.export    # workspace `billing` @ tag v4.1.0
        ref: v4.1.0

actions:
  nightly-2-3:
    type: task
    task: nightly                 # own task @ commit
    ref: 3f2a9c0e1b2c3d4e5f60718293a4b5c6d7e8f901

triggers:
  nightly-billing:
    type: scheduler
    cron: "0 2 * * *"
    task: billing.nightly         # a trigger may name another workspace's task
    ref: v4.1.0
```

Cutting a release is a pull request to `main` that bumps a `ref:`. Cutting
`release/2.5` needs no server configuration change.

:::caution[Before you merge YAML that uses `ref:`]
Every server replica must run a release with this feature first. An older
replica drops the unknown `ref:` key and silently runs the default branch.
See [Upgrading](#upgrading).
:::

## Where `ref` is allowed

| Place | `ref` qualifies |
|---|---|
| Flow step `action:` | the action |
| `type: task` action `task:` | the task |
| Scheduler and webhook trigger `task:` | the task |

`ref:` is **rejected**, never silently ignored, on:

| Place | `stroem validate` | At runtime |
|---|---|---|
| A hook (`on_success`, `on_error`, `on_cancel`, `on_suspended`), or a hook whose action is a `type: task` action carrying `ref` | error: "`ref` is not supported on hooks yet" | the hook is not fired; the error is logged to the job that fired it |
| An event-source trigger | error: "`ref` is not supported on event_source triggers yet" | the consumer is not started; a warning is logged on every reconcile |
| An agent `tools: [{task: …}]` entry | error: "`ref` is not supported on agent task tools yet" | the tool call is rejected with `400` |
| A flow step whose ref'd action is `type: agent` | — | `400`: "action '…': agent actions cannot be referenced with `ref` yet" |

`ref:` on an action of any type other than `task` is a `stroem validate`
error. The server does not validate workspace YAML when it loads it, so
there such a `ref:` has no effect — run `stroem validate` in CI.

A ref is a fixed string, or a template on a flow step, a `type: task`
action or a webhook trigger — see [Templated refs](#templated-refs).

## Templated refs

A `ref:` can be a Tera template, so the person who runs a task, the system
that calls a webhook, or an earlier step chooses the branch, tag or commit:

```yaml
tasks:
  deploy:
    input:
      branch: { type: string, default: main }
    flow:
      build:
        action: build
        ref: "{{ input.branch }}"          # chosen in the Run form
      find:
        action: find-release
      publish:
        action: billing.publish
        ref: "{{ find.output.branch }}"    # chosen by an earlier step
        depends_on: [find]

triggers:
  on-push:
    type: webhook
    name: on-push
    task: build
    ref: "{{ input.body.ref }}"            # chosen by the caller
```

A `ref:` is a template when it contains `{{` or `{%`.

| Place | Resolved | The template can read |
|---|---|---|
| Flow-step `ref:` | when the step becomes ready | `input` (job input), `state`, `global_state`, `job`, earlier steps' `output` / `error`, `each` (in a `for_each` instance) |
| `type: task` action `ref:` | when the step is dispatched | `input` (the step's `input:` rendered as strings, without the task's schema and without the action's input defaults), `job` |
| Webhook trigger `ref:` | when the webhook fires | `input` (`body`, `headers`, `method`, `query` and the trigger's `input` defaults) |

No ref template can read `secret` or use `vals`. A template is a
`stroem validate` error on scheduler triggers, hooks, event sources, agent
task tools and actions other than `type: task`.

**The owner.** The action name decides the owner workspace when the job is
created (`billing.publish` -> `billing`); the template chooses only the
commit. The action name itself is never a template. An owner that is not a
git workspace, an unknown workspace or a library is a `400` at job creation.

**When.** Until it is resolved, a flow step is *deferred*: job detail shows
`action_type: deferred` and the UI shows `ref: {{ … }}` on the step. A root
step (no `depends_on`, `when` or `for_each`) is resolved while the job is
created. Each `for_each` instance renders its own `each.item`.

**One commit per ref.** Inside one job, every step that uses the same owner
and the same ref value runs the same commit — static and templated refs
alike — even if the branch moves while the job runs. A task retry, a re-run
and a restart resolve again.

**An empty value means no ref.** The rendered value is trimmed. A value
that is empty, or only whitespace, means no ref: the step, the `type: task`
step or the webhook job runs as if the `ref:` key were absent. That is the
job's own commit in a pinned job, the workspace's live config in other jobs,
and the owner's live config for `ws.action`. A `type: task` action in a
pinned job keeps the job's commit, and a task that would call itself fails
the step, as without `ref:`. The step shows no ref. So an optional input can
choose the branch:

```yaml
tasks:
  deploy:
    input:
      branch: { type: string }
    flow:
      build:
        action: build
        ref: "{{ input.branch | default(value='') }}"   # blank: as without ref
```

The owner rules still apply to the template: the owner must be a git
workspace and cannot be a library, whatever the value. One difference
remains in a job without a pin: a step that becomes ready after the job
starts reads its action definition from the live config at that time, not at
job creation. For an action of the job's own workspace, its files stay those
of the job's revision.

:::caution[Upgrade every replica first]
A server replica that runs 0.18.3 or older fails a step whose templated ref
renders empty (`the rendered ref is empty`) and answers `400` to such a
webhook. Do not rely on an empty value until every server replica runs a
release with this change.
:::

**Errors.** These fail the step — never the job creation — with a message
that never shows the value:

| Rendered value | Step error |
|---|---|
| not a branch, tag or full SHA | `… is not a valid branch, tag or full commit SHA` |
| contains a secret value | `… contains a secret value` |
| no such branch or tag | `… was not found in workspace '…'` |
| the action or task is missing at that commit | `workspace '…' has no such action or task at the rendered ref` |
| git is unreachable | `workspace '…' is not available at the rendered ref yet` |
| the template fails to render | `the ref template could not be rendered: …` |

A failed resolution uses no retry attempt. A template syntax error or an
unknown filter is a `400` at job creation. A webhook answers `400` for these
errors (`500` when git is unreachable) and creates no job.

**`type: task` actions.** The ref reads the step's `input:` rendered as
strings, before the task is known. A string nested inside an object of the
step input must render too, even if the task never uses it, or the step
fails. `{% raw %}…{% endraw %}` makes such a string render, but the child job
receives the text with the `{% raw %}` tags: the child-input render copies a
nested object unchanged (and a `json` field refuses the tags). So literal
`{{` or `{%` text nested in an object cannot reach a task whose action has a
templated ref. The step input is rendered
twice — once for the ref, once for the child job — so a value that changes
between renders (`now()`, a `vals` lookup) can differ. A templated ref that
resolves to the same owner, the same task and the same commit as the running
job fails the step, like a static self-reference.

**Tools.** `stroem validate` checks a templated ref's syntax and skips the
action or task lookup with a warning. `stroem run` ignores `ref:` and runs
the local action.

:::caution[Who can choose the commit]
There is no allow-list. Anyone who can set a value the template reads — a
user with `Run` on the task, any caller of the webhook (anyone at all when it
has no `secret`), or an earlier step's output — can make the server run any
branch, tag or commit of the owner workspace with that commit's secrets and
connections. With push access to the owner repository, that runs any code.
Never pass a secret into a value a ref template reads: a rendered ref is an
identifier and is never masked. The server rejects a rendered ref that
contains a known secret value of 4 or more characters, but not a transformed
one (`| upper`, `| replace`).

Give a webhook with a templated `ref:` a `secret`. Without one, any caller
can make the server look up a ref that does not exist — one `ls-remote` of
the owner repository per unknown value, under the workspace's resolve lock.
:::

Before you merge YAML with a templated `ref:`, every server replica must run
a release that supports it — see
[Migrations 051 and 052](https://fremvaerk.github.io/stroem/operations/migration-051-052/).

## Ref forms

| Written | Meaning |
|---|---|
| 40 hex characters | that commit (normalised to lowercase) |
| `refs/heads/<name>` | branch `<name>` |
| `refs/tags/<name>` | tag `<name>` (an annotated tag resolves to its commit) |
| any other `<name>` | branch `<name>` if it exists, else tag `<name>` |

A bare name must be a valid git branch name. Other `refs/…` namespaces
(`refs/pull/…`, `refs/notes/…`) are not supported. Short SHAs are not
supported either: a 4–39 character hex string that names no branch or tag
fails with "short commit SHAs are not supported; use the full 40-character
SHA".

## How a reference resolves

The **owner** of the referenced name is decided first. Then the name is
looked up in the owner's config **at the ref**:

| Written | Resolves to |
|---|---|
| unqualified name + `ref` | your own workspace at the ref |
| `ws.name` + `ref` | workspace `ws` at the ref |
| unqualified name, no `ref`, inside a pinned config | the same commit (the pin is inherited) |
| `ws.name`, no `ref`, inside a pinned config | `ws`'s live config (the [cross-workspace](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/) rule) |
| a library item + `ref` | `400` — libraries are server-level and have no ref |
| `ref` on a folder workspace | `400` — only git workspaces have refs |
| `ws.name` + `ref`, `ws` not configured | `400` — unknown workspace |

With `ref`, the owner is decided from the name alone, so the name does not
have to exist on your default branch: a step may call an action that exists
only on the release branch. Inside a release, an unqualified name resolves
inside that release: `nightly` at `release/2.3` calling `action: helper`
gets 2.3's `helper`.

The owner does not have to be healthy. A ref can be resolved while the
owner's default branch fails to load, because refs use their own copy of the
repository (see [Storage](#storage-on-the-server)).

A ref'd target runs **in the owner's context at that commit**: its files,
its `secrets:`, its connections. Access control, the `shared: true`
connection gate and error withholding key on the **workspace name**, never
the ref. `own@release/2.3` is your own workspace; `billing@v4.1.0` is
cross-workspace, exactly like `billing`.

A task may call **itself** at another ref: the same task at another commit
is a different version. Nesting is bounded by the maximum task depth (10),
as for any other `type: task` chain.

## Pinning

Every ref in a job resolves to a commit **once**, when that job is created:

- a flow step's action, and a `type: task` step's task, are pinned when the
  job containing them is created;
- a trigger resolves its ref when it fires;
- a child job resolves its own nested refs when it is created.

A branch that moves mid-job therefore cannot split one job across two
commits of the same ref.

A job that runs on a ref is a **pinned job**: a ref'd `type: task` child, a
job created by a trigger with `ref:`, or a job derived from a pinned job
(below). Its flow, inputs, hooks, retries and
approvals all come from that one commit for its whole life, even if the
branch moves meanwhile.

| Derived job | Runs at |
|---|---|
| `type: task` child with its own `ref` | that ref, resolved when the parent was created |
| `type: task` child without `ref`, whose task is in the same workspace as the pinned config its action is written in | that config's commit (inherited) |
| `type: task` child without `ref`, whose task is in another workspace (or whose action is a live foreign action) | that workspace's live config |
| Hook job of a pinned job | the pinned job's ref and commit; the hook comes from that commit |
| Task-level retry of a pinned job | the same ref and commit |
| Agent task-tool child of a pinned job | the same ref and commit |
| Re-run or Restart of a pinned job | the ref **re-resolved** — a branch moves to its current tip (see [Re-run and Restart](#re-run-and-restart)) |

In templates, `{{ job.revision }}` of a pinned job is its commit and
`{{ job.ref }}` the ref as written. `{{ job.ref }}` renders as an empty
string for every other job. Hooks get the same value as `hook.ref`. Over the
API, a pinned job carries `ref`, and steps carry `action_ref`,
`task_workspace`, `task_ref` and `task_revision` (see the
[API reference](https://fremvaerk.github.io/stroem/reference/api/#get-job-detail)). The UI shows a
`@ release/2.3 · 3f2a9c0` badge on the job and on every step that carries a
pin.

## Freshness

Branches and tags are listed with one `ls-remote` per workspace. The listing
is cached for the owner workspace's `poll_interval_secs`, the same freshness
as the default branch.

| Situation | What happens |
|---|---|
| Push to `release/2.3`, next run 2 minutes later | the new commit |
| Push, next run 20 seconds later | may still get the previous commit |
| A branch or tag that was not in the cached listing | the listing is refreshed at once, so a new branch is found immediately |
| A job is running when the push lands | it keeps its commit |
| The branch is deleted | resolution fails once the cached listing expires (`400` / trigger MISSED) |
| The git server is unreachable | the last listing is used, with a warning; with none, the run fails (`500` / trigger MISSED) |

Each server replica keeps its own listing, so two jobs created on different
replicas within one `poll_interval_secs` window may resolve a branch to
different commits.

Nothing is loaded ahead of time. The first run after a push, and the first
run after every server restart, fetches, checks out and loads that commit.
That can take seconds for a large repository.

## Running pinned steps

### Claim

A worker's claim can land on a replica that has not loaded the step's commit
yet (another replica created the job, or the server restarted). The claim
then loads the commit itself, for at most
[`pin_store.claim_load_budget_secs`](https://fremvaerk.github.io/stroem/getting-started/configuration/#pin_store)
(default 20 seconds). Keep that budget below the workers'
`request_timeout_secs` (30 seconds by default).

If the load does not finish within the budget, or the git server is
unreachable, the claim is **released**: the step goes back to `ready` and is
offered again 10 seconds later. The load continues in the background, so the
next claim usually finds the commit ready. The job log shows
`[pin] etl@release/2.3 (3f2a9c0) not available yet on this server, retrying: …`.
A release is not a failure: it uses none of the step's retry attempts.
After 30 releases the step fails with
`[pin] … still unavailable after 30 attempts: …`, and step retry applies as
usual. Each release cycle takes up to the claim budget, plus the 10-second
release delay, plus the time until a worker polls again — 30 × (budget + 10 s
+ poll). With the defaults that is a little over 5 minutes when the git
server fails fast, and about 15 minutes or more when it hangs until the
budget runs out.

A permanent problem (the commit was force-pushed away, or its YAML does not
load) fails the step at once with `[pin] … cannot be loaded: …`.

### Files

The worker downloads the step's files at the pinned commit. The server
builds that tarball from a clean checkout of the commit, which has no `.git`
directory. If the server cannot reach the git server for a commit it has not
loaded, it answers `503` with `Retry-After: 5`, and the worker tries again
every 5 seconds — up to 12 attempts in all, about a minute — before it fails
the step.

A `runner: pod` step's pod downloads the files itself, in its
`workspace-init` container. It asks for the same commit and retries a `503`
the same way. Workers released before this was fixed asked for the live
workspace instead, so a pod step ran the **default branch's files** with the
pinned commit's definitions (env, `source:` path). Upgrade the workers to
fix it; the server needs no change.

### Settlement

Moving a pinned job forward after a step (promoting the next steps, firing
hooks, scheduling a retry) needs the commit's definitions. If they cannot be
loaded at that moment, the job waits, with
`[pin] … not available yet: …` in its log. The recovery sweep re-advances
such a job on every pass (every `recovery.sweep_interval_secs`, 60 seconds
by default), so it continues once the git server is reachable again. Each
pass handles a bounded batch (at most 50 jobs and about a minute of work,
skipping other jobs of a commit it just failed to load) and the next pass
continues where it stopped, so with many waiting jobs during a long outage
some are picked up a few passes later. That
includes a job that never started because its step failed at claim (after
30 releases) or its files could not be downloaded: once the definitions
load, the sweep skips the steps that depended on it and settles the job.

If the commit can never load again (force-pushed away, or the YAML at it is
broken), the job is **failed** with `[pin] … cannot be loaded: …`, and its
steps that have not started yet are cancelled. Steps already running keep
running — their workers are not told to stop — and finish on their own. Its
hooks and task-level retry do not run, because their definitions are
unreadable.

## Task state

A pinned job's [task state](https://fremvaerk.github.io/stroem/guides/task-state/) and global state are kept
**per ref string**: `release/2.3` and `release/2.4` never overwrite each
other, nor the default branch's state. Bumping `ref: v2.3.1` → `ref: v2.3.2`
starts with empty state. Manual state uploads always write the default
(unpinned) partition.

## Access control

A pinned job is authorised by the folder its **own commit** declares for its
task (`task_folder` on the job), never by the live task's folder, even when
a task of the same name exists on the default branch. Two refs of one task
that declare different folders are therefore authorised independently, in
job detail, logs, artifacts, the log stream, worker detail, the job list
and over MCP. See [Authorization](https://fremvaerk.github.io/stroem/operations/authorization/#jobs-and-task-folders).

Task duration statistics (percentiles, ETA) describe the live task only:
pinned jobs are left out.

## Secrets and redaction

A commit's secrets can differ from the default branch's, and can exist only
at that commit. The API responses that show a job's input, output and step
errors mask secrets with the live workspaces' values **plus** the secret
values of every pinned commit the job is connected to:

- the commits of the job and its steps;
- the commits of every job in the same job tree (parent, children,
  siblings), because values are copied between them;
- the commits of the jobs it was made from — the job that fired a hook, the
  job a restart was restarted from, the job a re-run re-ran (a re-run of a
  task retry replays that retry's resolved input), the job a task retry
  retries — and of their job trees. Chains of restarts and re-runs are
  followed up to 32 links; a longer chain masks everything.

These responses are job detail (`GET /api/jobs/{id}`), the recent steps on
worker detail, the sync webhook response and the webhook job-status poll,
and MCP `get_job_status`.

Job **logs** (REST, the WebSocket stream, MCP `get_job_logs`) and
**artifacts** are not masked at all: a secret that exists only at a ref and
that a script prints appears in the log exactly as printed. See
[Secrets & Encryption](https://fremvaerk.github.io/stroem/guides/secrets/#api-redaction).

When some of the secret values those responses need cannot be loaded, the
responses never fall back to a partial mask:

| Situation | Job detail, webhook, MCP | Worker detail |
|---|---|---|
| A commit cannot be loaded right now (git server unreachable on this replica) | `503` "redaction set unavailable, retry" (webhook responses keep `job_id`; MCP returns an error) | that row's `error_message` is `••••••` |
| A commit can never load (force-pushed away, broken YAML, secrets that have not loaded for an hour), or the job is connected to too many jobs to check | `200`, with every content string masked as `••••••`; ids, statuses and timestamps stay | that row's `error_message` is `••••••` |

The price is that a cold commit anywhere in a job's tree can make an
unrelated job's detail answer `503` during a git outage.

Errors are handled the same way as for
[cross-workspace references](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/#trust-model-for-owner-side-rendering-errors):
an error rendering the **owner's** templates of a ref'd action or task in
another workspace is withheld from the caller, with a fixed "details
withheld" message, and the full error goes to the server log. A commit whose
YAML does not load is reported only as
`[pin] ws@ref (sha) cannot be loaded: its configuration does not load`,
because the loader's message can quote secret values.

### Secrets that no longer decrypt

A commit's `sops` files and `vals` references are read when the server
loads that commit. If they fail — the `sops` key or the KMS key is not
available, a `vals` backend is down, a `vals` path no longer exists — the
commit counts as **not available yet**, like a git outage: claims are
released and retried, a job waits, the responses above answer `503`. The
server remembers such a failure for 30 seconds, so `sops` and `vals` run at
most twice a minute per commit, not on every request.

Old commits often stay undecryptable for good: key rotation re-encrypts only
new commits, and a retired KMS key or a deleted `vals` path does not come
back. So once a commit's secrets have failed for **one hour** on a server
(counted from the first failure that server saw, and reset by any
successful load of that commit), the server treats the commit as one that
can never load: claims fail the step, a waiting job fails, and the
responses above answer `200` with everything masked. The server log names
the cause (`its secrets have failed to load for …s …; treated as
permanent`). Each replica counts on its own, and a restart starts over.

## Errors

| Condition | Execute API, webhook | Scheduler trigger | When a step runs |
|---|---|---|---|
| Unknown workspace, folder owner, library item + `ref`, invalid or templated ref | `400` | MISSED | — (the job is never created) |
| Branch or tag not found | `400` | MISSED | — (resolved at creation) |
| Name missing at that commit ("has no action … at ref …" / "has no task … at ref …") | `400` | MISSED | — (checked at creation) |
| The YAML at that commit does not load | `400`, fixed message | MISSED | step fails, fixed message |
| Commit no longer exists (force-pushed away) | `400` | MISSED | step fails; a pinned job fails |
| Git server unreachable with nothing cached, load timeout, `sops`/`vals` failure | `500` | MISSED | claim released and retried, up to 30 times (see [Claim](#claim)) |
| The commit's secrets (`sops`/`vals`) have failed to load for an hour on that server | `400`, fixed message | MISSED | step fails, fixed message; a pinned job fails |
| `ref` on an agent action | `400` | MISSED | — |

A `type: task` step whose pinned action or task cannot be loaded when it is
dispatched fails, even when the cause is transient: dispatch has no
release-and-retry path. This needs both a replica that has never loaded the
commit and a git outage.

A trigger fire that cannot resolve its target is logged as
`Trigger '<ws>/<name>' MISSED: …` and has no other effect: no
`cancel_previous`, no skipped row. It is not replayed.

## Triggers

A scheduler or webhook trigger's `task` may name another workspace's task
(`ws.task`), with or without `ref`. The job is created in the task's owner
workspace, at the ref's commit when there is one. The ref is resolved when
the trigger fires, before the concurrency policy. The concurrency policy is
keyed on the **defining** workspace's trigger (`source_id` =
`<defining ws>/<trigger>`), and a `skip` row is recorded in the task's
owner workspace with the resolved ref and commit. `triggers: false` on the
defining workspace still suppresses the trigger.

A webhook with `force_refresh: true` reloads its workspace and is then
matched and authenticated again against the refreshed definition, so a
refresh that changes the `ref` runs the new one (see
[Webhook API](https://fremvaerk.github.io/stroem/reference/webhook-api/)).

## Re-run and Restart

Re-running or restarting a pinned job re-resolves its ref **before** the
task is looked up: a branch moves to its current tip, a tag or SHA stays
put. The task is looked up at that commit, so it may exist only on that
branch, and the new job is pinned to the new commit with the same ref.
Carried-over steps of a restart keep the output they produced at the old
commit; the restarted steps run at the new one.

| Situation | Answer |
|---|---|
| No access to the source job's folder, or to the folder the task declares at the re-resolved commit | `404` |
| `View` only on either folder (both need `Run` on both) | `403` |
| Re-run posted to a different task name than the source's | `400` "Source job … is a run of task '…', not '…'" |
| The task no longer exists at the re-resolved commit | `400` "Task '…' does not exist at ref '…' (sha)" |
| The ref no longer exists | `400` |
| Git server unreachable | `500` |

Access is checked twice: on the source job's folder (its `task_folder`)
first, then on the folder the task declares at the re-resolved commit — the
folder the new job will carry. Moving a task into a stricter folder on its
branch therefore also restricts who can re-run or restart its older jobs. A
restart dry run of a pinned job also contacts the git server, so the preview
can answer `500` during an outage. The UI hides Re-run and Restart for a pinned
job whose task does not exist on the default branch; use the API for those
(`POST /api/workspaces/{ws}/tasks/{task}/execute` with `source_job_id`, or
`POST /api/jobs/{id}/restart`).

## Storage on the server

Each server replica keeps its own **pin store**: one bare clone per git
workspace, never the clone the default branch is loaded from, plus one
read-only checkout and one loaded config per commit in use. Commits that no
active job needs are evicted, beyond the `keep_recent_per_workspace` most
recently used ones. The bare clones are never garbage-collected. See
[`pin_store`](https://fremvaerk.github.io/stroem/getting-started/configuration/#pin_store) for the settings,
and [Metrics](https://fremvaerk.github.io/stroem/operations/metrics/#pin-store) for `stroem_pin_loads_total`
and `stroem_pins_cached`.

Loading a commit uses the
[`workspace_reload`](https://fremvaerk.github.io/stroem/getting-started/configuration/#workspace_reload)
budgets: `peek_timeout_secs` for the branch and tag listing, and
`load_timeout_secs` for fetch, checkout and config load together. At most
four commits load at once per replica, separately from the default-branch
watchers.

## Security

There is no allow-list. Anyone who can merge a `ref:` into **any** workspace
can make the server run **any** branch, tag or commit of **any** configured
git workspace with that workspace's secrets. That includes resolving
whatever `vals` references an unreviewed branch contains. Restrict who can
merge to your workspaces' default branches accordingly.

## Upgrading

**Do not merge YAML that uses `ref:` until every server replica runs a
release with this feature.** An older replica ignores the unknown `ref:` key
and silently runs the default branch instead.

**An empty templated ref** means no ref only on a release after 0.18.3. An
older replica fails the step (`the rendered ref is empty`), and a webhook
answers `400`. Upgrade every server replica before a workspace relies on an
empty value.

The same release changes some behaviour for **every** job, including jobs
that use no `ref:` at all: redaction of webhook, MCP and worker-detail
output, webhook re-authentication after `force_refresh`, task state of
cross-workspace steps, and how old revisions' files are served. Read
[Migrations 049 and 050](https://fremvaerk.github.io/stroem/operations/migration-049-050/) before upgrading.

## Not yet supported

- A templated `ref` on scheduler triggers, hooks, event sources and agent
  task tools; an allow-list of the refs a template may choose.
- A per-run ref override (testing a feature branch from the UI, API, MCP or
  CLI).
- `ref` on hooks, event sources, agent actions and agent task tools.
- Libraries at a ref.
- Manual state upload to a ref's partition.
- Re-run and Restart from the UI of a pinned job whose task does not exist
  on the default branch (the API works).

---

## Artifacts

Artifacts let a task produce files — reports, screenshots, build outputs — that
humans can view or download from the UI, CLI, or MCP after the job completes.

## Producing artifacts

Write files to `/artifacts/` (or the `$ARTIFACTS_DIR` env var for portability):

```yaml
name: nightly-report
steps:
  - name: build
    action:
      type: script
      runner: local
      language: shell
      script: |
        ./tools/report --out "$ARTIFACTS_DIR/report.html"
        cp build/dist.zip "$ARTIFACTS_DIR/"
```

When the step succeeds, the worker uploads every file in `/artifacts/` (recursively). Failed or cancelled steps discard their `/artifacts/` directory.

### What gets uploaded

- Regular files (recursive — `reports/q1.html` keeps the slash in its name).
- Dotfiles (e.g. `.config`). **Don't put `.env` here.**
- Empty files.
- Symbolic links are skipped + warned.
- Filenames with `..`, null bytes, or control characters are rejected + warned.

### Limits

| | Default | Configurable |
|---|---:|---|
| Per-file | 100 MiB | `artifact_storage.max_file_bytes` |
| Per-job  | 1 GiB   | `artifact_storage.max_job_bytes` |

Exceeding either fails the step loudly.

### Runner support

| Runner | Supported |
|---|---|
| Shell (local) | Yes |
| Docker (`script:docker`) | Yes |
| Docker (`type:docker`) | Yes |
| Kubernetes (any) | No (deferred — same limitation as state `/state-out`) |
| Agent / Approval / Task actions | n/a |

On unsupported runners, `$ARTIFACTS_DIR` is unset; scripts that try `> "$ARTIFACTS_DIR/foo"` will fail loudly with "ambiguous redirect".

## Viewing & downloading

UI: a new **Artifacts** section appears near the top of the Job Detail page. Safe MIME types (PNG, JPEG, GIF, WebP, plain text) open inline in a new tab. Everything else — including HTML and SVG (for XSS hardening) — downloads.

CLI:

```sh
stroem-api artifacts list <job-id>
stroem-api artifacts download <job-id> report.html -o ./report.html
```

Hooks: `hook.artifacts` is a list of `{name, content_type, size_bytes, url, step_name, created_at}`. Example Slack notification:

```yaml
on_success:
  - action: slack-notify
    input:
      text: |
        Job done. Outputs:
        {% for a in hook.artifacts %}- <{{ a.url }}|{{ a.name }}> ({{ a.size_bytes }} B)\n{% endfor %}
```

MCP: `list_artifacts(job_id)` and `get_artifact(job_id, name)`. `get_artifact` returns text/JSON/YAML/XML as inline text, `image/*` as image blocks the agent can view, and other binary (PDF, archives, …) as a base64 blob resource — all capped at 1 MiB. Larger artifacts must be fetched via the HTTP artifact URL from `list_artifacts` rather than embedded in the agent's context.

**NOTE:** Artifacts are files that steps produce. To give a file *to* a task, see [File Inputs](https://fremvaerk.github.io/stroem/guides/file-inputs/).
## Retention

Artifacts live as long as the job row does. Once a job is deleted (via the existing `retention.job_days` sweep), its blobs and rows go with it. There is no separate artifact TTL today.

## Security notes

Artifacts produced by a workflow are served with the stored Content-Type, but anything that could execute in the browser (HTML, SVG, XML) is force-downloaded rather than rendered. `X-Content-Type-Options: nosniff` blocks browsers from second-guessing.

`View` permission on the task is sufficient to list and download artifacts — the same permission required to view logs. Don't put secrets in `/artifacts/`.

---

## File Inputs

A `type: file` input field lets a person or a program give a task a file. The step does not get the content in a template value. It gets a **path** to a read-only copy of the file, for example as a command-line argument.

:::caution[Rollout order]
An older server replica treats `type: file` as a [connection type](https://fremvaerk.github.io/stroem/guides/connections/) and answers `400` for such a task. Upgrade **every server replica first, then every worker, then** add YAML with `type: file`. See [Migration 054](https://fremvaerk.github.io/stroem/operations/migration-054/).
:::

## Declaring a file field

```yaml
actions:
  show-file:
    type: script
    script: |
      sha256sum "{{ input.data_file }}" && cat "{{ input.data_file }}"
    input:
      data_file: { type: file, required: true }

tasks:
  file-demo:
    mode: distributed
    description: Demonstrates the file input type.
    input:
      data_file:
        type: file
        required: true
        description: Any file
    flow:
      show:
        action: show-file
        input:
          data_file: "{{ input.data_file }}"
```

In a template, `{{ input.data_file }}` renders to a path such as `/inputs/file_7f3a91c2…/report.csv`. The file is read-only. The stored value of the field (in the job input, the API and hook payloads) is the **file ID**, never the path.

`type: file` is allowed in a task `input` and in an action `input`. `stroem validate` rejects:

- a `file` field that has `default`, `options`, `allow_custom`, `multiple` or `secret`;
- a `file` property in a [connection type](https://fremvaerk.github.io/stroem/guides/connections/), and a connection type named `file` (the name is reserved);
- a `file` field in an `approval` action;
- an agent action whose `tools` name a task with a `required` file field (an agent cannot supply a file).

It warns when a task with a `required` file field is the target of a scheduler trigger, an event source trigger or a webhook in the same workspace (a trigger with `ref:` is not checked), because none of them can supply a file.

The server does not run these checks when it loads a workspace. A `default` on a `file` field that reaches the server is not attached to the job: the template gets the literal default, and the step gets no file.

A `required` file field with no value fails job creation in every creation mode. This differs from other types, which the server does not check.

### Passing a file to an action or a child task

A flow step gives the file on with a template:

- `input: { path: "{{ input.data_file }}" }` gives a `string` action field the path.
- `input: { data: "{{ input.data_file }}" }` gives a `file` action field the same file. In the action body `{{ input.data }}` is the same path.

An action `file` field also accepts a file ID, or a path of a file that belongs to the same job. Any other value fails the step with `Input field 'data': the file cannot be used`.

A `type: task` step passes the file to the child job's `file` field in the same way, in the same or in another workspace. Only a child `file` field attaches the file to the child job: a child field of another type that gets `{{ input.data_file }}` receives the path `/inputs/{file_id}/{name}` as text, and the child's steps cannot read that path. A job input file field with the same name as an unset action field is copied by the target type: the file ID into a `file` action field, the path into any other field.

Re-run, Restart and [task retry](https://fremvaerk.github.io/stroem/guides/retry/) keep the file of the source job. Hook jobs and agent task-tool children do not get it. The tool schema that an agent sees for a task leaves out the task's `file` fields; a tool call for a task with a `required` file field fails with `Input field '…': a file is required`.

## Supply a file

Every client works in two calls: **upload** the file (you get a file ID), then **execute** the task with the ID as the field value.

### UI

The Run form shows a file picker for a `file` field. The browser uploads the file when you select it, with a progress bar. The form shows the file name and size, and offers **Remove** and **Replace**. **Run** is disabled while an upload runs or after it fails. A failed upload offers **Remove**. **Cancel** of a **Replace**, and **Remove** after a failed **Replace**, return to the file the field had before.

On the job page, the **Input files** card lists the files of the job with a download link.

On [Re-run](https://fremvaerk.github.io/stroem/guides/rerun-and-restart/#files), the field shows the file of the source job. Keep it, or choose **Replace** to upload another file. If you cancel the **Replace**, or remove a failed replacement, the field shows the file of the source job again.

### REST API

```sh
# 1. Upload the raw bytes (no multipart)
curl -X POST \
  -H "Authorization: Bearer $STROEM_TOKEN" \
  -H "Content-Type: text/csv" \
  --data-binary @report.csv \
  "http://localhost:8080/api/workspaces/default/tasks/import-report/files?name=report.csv"
# {"file_id":"file_7f3a…","name":"report.csv","size_bytes":1234,"sha256":"…",
#  "content_type":"text/csv","expires_at":"2026-10-31T12:00:00Z"}

# 2. Execute with the file ID
curl -X POST \
  -H "Authorization: Bearer $STROEM_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": {"data_file": "file_7f3a…"}}' \
  http://localhost:8080/api/workspaces/default/tasks/import-report/execute
```

See the [API reference](https://fremvaerk.github.io/stroem/reference/api/#upload-input-file). You can upload once and execute several times within the upload lifetime.

### CLI

```sh
stroem-api trigger import-report --file data_file=./report.csv
```

`--file FIELD=PATH` can be repeated. The CLI uploads each file, then executes the task with the IDs. A key that you give in both `--input` and `--file` is an error.

`stroem run` runs a task locally without a server. The value of a `file` field is a local path. The CLI checks that it is a regular file and copies it read-only to `{temp}/stroem-run-inputs-{uuid}/{file_id}/{safe_name}`, so the path in the script has the same shape as on the server. Only your user can open `stroem-run-inputs-{uuid}` (mode `0700`), so other users of the host cannot read the copy. The directory is removed when the run ends:

```sh
stroem run import-report --input '{"data_file": "./report.csv"}'
```

`stroem run` still runs only `type: script` with `runner: local`.

### MCP

An agent has two ways to send a file with `execute_task`:

- **Inline.** The `file` field is an object `{"name": "report.csv", "content": "<utf-8 text>"}` or `{"name": "report.csv", "content_base64": "<base64>"}`, with exactly these two keys. Any other shape, or base64 that does not decode, gives `Input field 'f': the file cannot be used`. The decoded size must not be more than 1 MiB (`input_files.mcp_inline_max_bytes`). A larger file gives `Input field 'f': inline content is larger than N bytes`. The server stores the file as an upload of the caller (content type `application/octet-stream`), then creates the job.
- **Upload URL.** Call `create_upload_url(workspace, task_name, file_name)`. It needs the Run permission on the task, a task with a `file` field, and a usable file name. It returns `{upload_url, method: "POST", max_bytes, expires_at}`. `POST` the raw bytes to the URL **without** a Bearer header: the URL token is the credential. The URL is valid for 15 minutes (`input_files.upload_url_ttl_secs`) and works **once**: the token is used up when the request arrives, before the body is read, so if an upload fails, ask for a new URL. A used, expired or unknown URL gives `404 Upload URL not found`. The response is the same as for the REST upload, and the upload belongs to the user who asked for the URL. Then call `execute_task` with the returned `file_id`.

`create_upload_url` needs `auth.base_url` in the server config. Without it the tool answers `upload URLs need auth.base_url; send the file inline or use the REST upload`. See the [MCP guide](https://fremvaerk.github.io/stroem/guides/mcp/).

## Where the file is in the step

The path is `{root}/{file_id}/{safe_name}`. The path depends on the file only, never on the field that holds it, so one file has one path in every template of a step, and two files never share a path.

| Runner | Root |
|---|---|
| `runner: local` | `{temp}/stroem-inputs-{uuid}/in`, where `{temp}` is the worker's temporary directory (`TMPDIR`). New for each claim and removed after the step. The parent `stroem-inputs-{uuid}` has mode `0700`, so other users of the host cannot reach the files |
| `runner: docker`, `type: docker` | `/inputs` (read-only bind mount) |
| `runner: pod`, `type: pod` | `/inputs` (read-only volume) |

In a context that no worker renders (an agent prompt, a `when` or `for_each` expression, a `type: task` step input, an approval message, a [templated ref](https://fremvaerk.github.io/stroem/guides/git-refs/)), the path uses `/inputs`. To test whether a file was given to an optional field, write `when: "{{ input.data_file | default(value='') }}"`: an absent field is a missing key, and a missing key is a template error that fails the step. A `for_each` item that holds a file path gets the step's own root at claim, so `{{ each.item }}` is a valid path on every runner.

On a pod, a separate init container `stroem-inputs-init` downloads each file from the server to an `emptyDir` volume `stroem-inputs`, then checks the size and the SHA-256. The main container mounts the volume read-only at `/inputs`. The init container uses the same image and retry policy as `workspace-init`, and is added only when the step has files. If a download or a check fails, the pod does not start the main container and the step fails. A pod manifest override can rewrite both names, as for `workspace-init`; Strøm does not validate that. For the other runners, the worker downloads the file, streams it to disk, and checks the size and SHA-256 before the runner starts. The file has mode `0444` and its directories `0555`, so a Docker step that runs as another user can still read it. A `503` from the server is retried every 5 seconds, up to 12 attempts.

A local step sees a path such as `/tmp/stroem-inputs-5d0c…/in/file_7f3a…/report.csv`. A retried step is claimed again with a new directory, and its paths are rendered again for that directory.

### Every step gets every file

**Every runner step of a job receives every file of that job**, not only the files the step uses. A step that does not mention a file still finds it in `/inputs` (or in the local root). Agent steps (`type: agent`) have no runner and receive no files.

This has consequences:

- A step that runs an action from **another workspace** ([cross-workspace reference](https://fremvaerk.github.io/stroem/guides/cross-workspace-references/)) receives all files of the caller job. The script of that workspace can list and read them. If you use another workspace's action in a task that has `file` fields, you trust that action with those files.
- A file that you give to a `type: task` child in another workspace can be downloaded by everyone who has View permission on the child job.
- Every instance of a [`for_each`](https://fremvaerk.github.io/stroem/guides/loops/) step downloads every file. On a pod runner, each instance pod runs its own init container.

## Limits, lifetime and names

| Setting | Default |
|---|---|
| Largest file (`input_files.max_file_bytes`) | 100 MiB. A larger upload gives `413` and nothing is kept |
| Lifetime of an unused upload (`input_files.upload_ttl_secs`) | 24 hours |
| MCP upload URL lifetime (`input_files.upload_url_ttl_secs`) | 15 minutes, single use |
| Largest inline MCP file (`input_files.mcp_inline_max_bytes`) | 1 MiB decoded |

See [Storage](https://fremvaerk.github.io/stroem/operations/storage/#input-files) for the configuration.

**Who can use an upload.** An upload belongs to the user who made it (an API key or an MCP OAuth token counts as its user) and to one task. A direct start of the task can use it: the UI, the REST API, `stroem-api`, MCP `execute_task`, Re-run and Restart. The upload must be younger than the lifetime above. With authentication off, any caller can use an upload. A file that is already attached to an earlier job can also be reused by the jobs that derive from it (Re-run, Restart, task retry and `type: task` children), with no time limit. A trigger, a webhook, an event source or a hook cannot use an upload. Any other case fails with `Input field 'data_file': the file cannot be used`. The message never contains the ID, the name or the path.

**Safe name.** The step sees a safe version of the file name. The original name stays on the job page and in downloads. The rule:

- Keep the base name only.
- Replace every character outside `A-Z a-z 0-9 . _ -` with `_`.
- Replace a leading `.` or `-` with `_`.
- Keep at most 128 bytes, and keep the extension.
- A missing or empty `name`, or a name that is `.` or `..` (or ends in `/`), gives `400 invalid file name`.

| Original name | Safe name |
|---|---|
| `report.csv` | `report.csv` |
| `x;rm -rf ~.csv` | `x_rm_-rf__.csv` |
| `-rf` | `_rf` |
| `.env` | `_env` |
| `my report (final).pdf` | `my_report__final_.pdf` |

The rule keeps the path safe to put into `script:`, `cmd:` and `args:`.

## What v1 does not support

- A webhook, a scheduler trigger, an event source, a hook or an agent task tool cannot supply a file. When the target task has a `required` file field, or the trigger's `input:` gives a `file` field a value, the scheduler logs `Trigger '…' MISSED` and a webhook answers `400`. The check runs after the target is resolved and before the concurrency policy, so nothing is skipped or cancelled. An event source cannot create a job for such a task, and a hook whose action or task needs a file is not fired (the error is logged to the job that fired it). An optional `file` field is simply absent in all these jobs.
- One file for each field.
- The file name, size and content type are not available in templates, only the path.
- The content of a file is not masked. Do not rely on secret masking for file content.
- `stroem run` runs local scripts only.

## Limitations

- **Helm with the Docker-in-Docker sidecar.** The `dind` sidecar does not share a volume with the worker. A `runner: docker` step sees an **empty** `/inputs` (and an empty `/workspace` and `/state`), because the daemon creates a missing bind source as an empty directory on its own file system. In such a deployment, use `runner: local` or `runner: pod` for steps that read input files.
- **Pod init failures.** When `stroem-inputs-init` fails, the step shows only the existing pod failure (exit code 1), with no text about the cause. The init container writes one line (the file ID and the check that failed) to its own log, which is not copied to the step log.
- **Large files on pods.** The init download counts toward the pod's 10-minute pending limit. Several large files, or a slow server, can reach it; the step then fails with `Pod stuck in Pending state …`.

## Retention

A file lives as long as a job uses it. When job retention deletes a job, the links of the job to its files go with it. A file that no job uses is deleted by the retention sweep once it is older than `upload_ttl_secs`, together with expired upload URLs. The sweep runs on every retention tick (`retention.interval_secs`, hourly by default), also when no `retention.job_days` or `worker_hours` is set. See [Retention](https://fremvaerk.github.io/stroem/operations/retention/#input-files).

## Troubleshooting

| Message | Cause |
|---|---|
| `Input field 'f': the file cannot be used` | The ID is unknown, belongs to another user or task, has expired or is being deleted, or the value is not a file ID |
| `Input field 'f': a file is required` | A `required` file field has no value |
| `invalid file name` | The `name` is missing or empty, or it is `.` or `..` |
| `task has no file input` (`400`) | The upload is for a task that declares no `file` field |
| `Source job not found` (`404`) | An upload with `source_job_id`: the job does not exist, or you may not view it |
| `could not load the job's input files` | The server could not read the job's files when a worker claimed the step (a database error). Step `retry:` applies |
| `file is larger than N bytes` (`413`) | The file is over `input_files.max_file_bytes` |
| `file storage is not configured` (`503`) | The server has no archive for blobs. It refuses uploads, and every run with a file value, also a Re-run or Restart of a job with a file. A step of a job that already has files fails when a worker claims it. See [Storage](https://fremvaerk.github.io/stroem/operations/storage/#input-files) |
| `this step has file inputs, which the worker does not support; upgrade the worker` | An old worker claimed a step of a job that has files. Upgrade the workers. Step `retry:` applies |
| `Input field 'f': inline content is larger than N bytes` | An MCP inline file is over the inline limit. Use `create_upload_url` |
| `upload URLs need auth.base_url; …` | Set `auth.base_url`, or send the file inline |
| `Upload URL not found` (`404`) | The URL was used, expired or never existed |

---

## MCP Integration

Strøm exposes a Model Context Protocol (MCP) endpoint that allows AI agents like Claude, Cursor, and other MCP-compatible tools to interact with your workflow orchestration server using a standardized interface.

## Overview

The MCP endpoint provides a set of tools for AI agents to:

- List and query available workspaces and tasks
- Retrieve detailed task definitions and execution history
- Execute tasks and monitor job status in real-time
- View logs and troubleshoot failed steps
- Cancel running or pending jobs

This enables AI agents to autonomously orchestrate complex workflows, answer questions about your infrastructure, and help debug issues.

## Enabling MCP

Add the `mcp` section to your `server-config.yaml`:

```yaml
mcp:
  enabled: true
```

The MCP endpoint is served at `http://localhost:8080/mcp` using Streamable HTTP transport (JSON response mode). When auth is configured on the server, MCP clients must send `Authorization: Bearer <token>` headers with each request.

**NOTE:** No additional configuration is required beyond enabling MCP. The endpoint automatically exposes all available workspaces, tasks, and jobs accessible to the authenticated user.
### Host allow-list (public deployments)

The MCP Streamable-HTTP transport validates the request `Host` header to prevent
DNS-rebinding attacks, accepting loopback hosts by default. Strøm automatically
allow-lists the host from `auth.base_url`, so a standard reverse-proxy deployment
works out of the box (this is also why `auth.base_url` is required when
`mcp.enabled: true`).

If the server is reachable under **additional** hostnames (an alias domain, a
second ingress), list them explicitly — otherwise requests to those hosts are
rejected before authentication with a `disallowed Host header` warning:

```yaml
mcp:
  enabled: true
  # Extra hostnames beyond auth.base_url's host and loopback. Include a port
  # only for non-default ports, e.g. "stroem.internal:8080".
  allowed_hosts:
    - alias.example.com
  # Browser Origin allow-list. Leave empty (default) for non-browser clients
  # like Claude Code; set it only when browser-based MCP clients connect.
  allowed_origins: []
```

## Available Tools

The MCP server provides eleven tools for interacting with Strøm:

| Tool | Description | Parameters |
|------|-------------|------------|
| `list_workspaces` | List all workspaces and their task counts, with the server-config `display_name` / `description` when set | none |
| `list_tasks` | List tasks in a workspace | `workspace?` (optional; lists all if omitted) |
| `get_task` | Get detailed task definition | `workspace`, `task_name` |
| `execute_task` | Execute a task and return job ID | `workspace`, `task_name`, `input?` |
| `get_job_status` | Get current job status with step details | `job_id` |
| `get_job_logs` | The end of a job's log (last 256 KiB), formatted | `job_id`, `step?`, `tail_bytes?` |
| `list_jobs` | List recent jobs with optional filters | `workspace?`, `task_name?`, `status?`, `limit?` |
| `cancel_job` | Cancel a running or pending job | `job_id` |
| `list_artifacts` | List a job's artifacts (name, content type, size, download URL) | `job_id` |
| `get_artifact` | Fetch one artifact's content | `job_id`, `name` |
| `create_upload_url` | Get a single-use URL to upload a file for a `file` input field | `workspace`, `task_name`, `file_name` |

When the log is longer than the tail, the result ends with `[truncated: showing the last 256.0 KiB of up to 83.3 MiB — earlier lines omitted]`.

`get_artifact` returns content shaped to the artifact's type: textual artifacts
(text, JSON, YAML, XML) come back as inline text, `image/*` artifacts as image
blocks the agent can view, and any other binary (PDF, archives, …) as a base64
blob resource. Artifacts larger than 1 MiB are not inlined — download them via
the HTTP artifact API (the URL is in `list_artifacts`) instead.

### Sending files

A task can have [`file` input fields](https://fremvaerk.github.io/stroem/guides/file-inputs/). `get_task` shows them as `type: file`. There are two ways to give such a field a file with `execute_task`.

**Inline**, for small files. Put an object in `input` instead of a value. The decoded content must not be more than 1 MiB (`input_files.mcp_inline_max_bytes`):

```
Call: execute_task(
  workspace="default",
  task_name="import-report",
  input={ "data_file": { "name": "report.csv", "content": "id,total\n1,10\n" } }
)
```

Use `content_base64` instead of `content` for binary data. The object must have exactly the two keys `name` and `content` (or `content_base64`); another shape or invalid base64 gives `Input field '…': the file cannot be used`. A file over the limit gives `Input field '…': inline content is larger than N bytes`. The server stores the file as your upload (content type `application/octet-stream`) and then creates the job.

**Upload URL**, for larger files and for agents that have a shell:

```
Call: create_upload_url(workspace="default", task_name="import-report", file_name="report.csv")
Returns: { "upload_url": "https://strom.example.com/api/uploads/…", "method": "POST",
           "max_bytes": 104857600, "expires_at": "2026-10-30T12:15:00Z" }
```

Then `POST` the raw bytes to `upload_url` with no `Authorization` header (`curl --data-binary @report.csv "$upload_url"`). The URL token is the credential. The URL works once and expires after 15 minutes (`input_files.upload_url_ttl_secs`). The token is used up when the request arrives, before the body is read: if the upload fails, ask for a new URL. A used, expired or unknown URL answers `404 Upload URL not found`. When the upload arrives, the server checks again that the URL's user may still run the task and that the task still has a `file` field. If not, it answers the same `404 Upload URL not found` and the URL stays used. The answer holds the `file_id`. Pass it as the field value in `execute_task`.

`create_upload_url` needs `auth.base_url` in the server config. Without it the tool answers `upload URLs need auth.base_url; send the file inline or use the REST upload`. It needs the Run permission on the task (View only gives `Insufficient permissions: execute requires Run access`), a task with a `file` field (else `task has no file input`) and a usable `file_name` (else `invalid file name`).

## Authentication

When auth is configured on your Strøm server, MCP clients must authenticate using the standard HTTP `Authorization` header:

```
Authorization: Bearer <token>
```

Strøm supports two token types:

- **JWT tokens**: Long-lived (30 days) user session tokens issued at login. Supported on both API keys and user logins.
- **API keys**: Long-lived programmatic tokens with the `strm_` prefix. Ideal for MCP integrations and CI/CD pipelines. Manage API keys via the Settings page or `/api/auth/api-keys` endpoint.

MCP clients without an `Authorization` header can still connect if auth is disabled on the server.

### OAuth 2.1 (MCP spec 2025-06-18)

When auth is enabled, Strøm acts as both the OAuth 2.1 **Resource Server** for `/mcp` and the **Authorization Server** that issues tokens for it. A spec-conformant MCP client connects with no manual configuration — it discovers the auth flow on the first 401.

The flow:

1. Client calls `/mcp` without a token.
2. Strøm responds `401 Unauthorized` with `WWW-Authenticate: Bearer resource_metadata="https://.../.well-known/oauth-protected-resource"`.
3. Client fetches that document — [RFC 9728](https://datatracker.ietf.org/doc/html/rfc9728) Protected Resource Metadata — which advertises the authorization server URL.
4. Client fetches `/.well-known/oauth-authorization-server` — [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414) — to learn the endpoints, grant types, and PKCE methods. Strøm advertises:
   - `response_types_supported: ["code"]`
   - `grant_types_supported: ["authorization_code", "refresh_token"]`
   - `code_challenge_methods_supported: ["S256"]` (OAuth 2.1 forbids `plain`)
   - `token_endpoint_auth_methods_supported: ["none", "client_secret_post"]`
5. Client self-registers via `POST /oauth/register` — [RFC 7591](https://datatracker.ietf.org/doc/html/rfc7591) Dynamic Client Registration. The response includes a `client_id` (and `client_secret` if the client requested confidential auth).
6. Client opens the user's browser to `/oauth/authorize?response_type=code&client_id=...&redirect_uri=...&code_challenge=...&code_challenge_method=S256&scope=mcp&resource=https://.../mcp&state=...`.
7. Strøm validates the request, then redirects to `/consent?...` — the React consent screen. If the user isn't logged in, the SPA bounces them through `/login?next=/consent?...` first.
8. The user clicks **Allow**. The SPA `POST /api/oauth/consent` mints a single-use authorization code and returns the final redirect URL, which the browser then navigates to: `redirect_uri?code=<code>&state=<state>`.
9. Client exchanges the code at `POST /oauth/token` with the matching `code_verifier`. Strøm verifies PKCE (`BASE64URL(SHA256(code_verifier)) == stored code_challenge`), then mints:
   - An **access token** — JWT, 1 hour TTL, `aud` bound to the canonical `/mcp` URL (RFC 8707 Resource Indicators).
   - A **refresh token** — opaque, DB-backed, 30 day TTL, **rotated** on each use (OAuth 2.1 §4.13).
10. Client uses the access token as the `Bearer` value on `/mcp`. The MCP middleware enforces the `aud` claim — a token issued for a different resource is rejected, even if signed with the same secret.

#### Dynamic Client Registration (DCR)

`POST /oauth/register` is public — any client can register without an admin step. To limit abuse:

- Each registration expires after **30 days**. Long-running integrations re-register on the first request after expiry; the cost is one automatic POST.
- Maximum 5 `redirect_uris` per client.
- Only `https://...` redirects are allowed, except for `http://localhost`, `http://127.0.0.1`, and `http://[::1]` (loopback exceptions per OAuth 2.1 §4.1).
- Only the `authorization_code` + `refresh_token` grants and the `mcp` scope are permitted.

#### Audience binding

Tokens issued by `/oauth/token` carry an `aud` claim equal to the canonical resource URL (e.g. `https://stroem.example.com/mcp`). The MCP middleware verifies this on every request. A token leaked from one resource server cannot be replayed against another, even if both trust the same issuer.

Legacy `strm_*` API keys and login-flow JWTs predate the OAuth flow and have no `aud` claim — the middleware accepts them unchanged. New integrations should prefer the OAuth flow.

#### Configuration

Set `auth.base_url` in `server-config.yaml` so the discovery documents and `aud` claim resolve to the canonical public URL:

```yaml
auth:
  jwt_secret: "..."
  refresh_secret: "..."
  base_url: "https://stroem.example.com"
```

Without `base_url`, Strøm falls back to the request `Host` (respecting `X-Forwarded-Host` / `X-Forwarded-Proto`) — fine for local dev, brittle behind multiple proxy layers.

#### Claude Desktop with auto-discovery

A spec-conformant MCP client only needs the `/mcp` URL. Auth handles itself:

```json
{
  "mcpServers": {
    "stroem": {
      "url": "https://stroem.example.com/mcp",
      "transport": "streamable-http"
    }
  }
}
```

The first connection triggers the 401 → discovery → DCR → browser consent → token exchange flow described above. After the user clicks **Allow** once, Claude Desktop refreshes the access token silently for as long as the refresh token is valid (30 days).

## Connecting from Claude Desktop

To connect Claude Desktop to your Strøm server, add the MCP configuration to `claude_desktop_config.json`:

On macOS/Linux, edit `~/.claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "stroem": {
      "url": "http://localhost:8080/mcp",
      "transport": "streamable-http"
    }
  }
}
```

On Windows, edit `%APPDATA%\Claude\claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "stroem": {
      "url": "http://localhost:8080/mcp",
      "transport": "streamable-http"
    }
  }
}
```

If your Strøm server requires authentication, add the token:

```json
{
  "mcpServers": {
    "stroem": {
      "url": "http://localhost:8080/mcp",
      "transport": "streamable-http",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_OR_JWT_TOKEN"
      }
    }
  }
}
```

Restart Claude Desktop after updating the config. The Strøm tools will appear in Claude's tool palette.

## Example: Typical Agent Workflow

Here's a typical interaction sequence between an AI agent and Strøm via MCP:

### 1. Discover available workspaces

```
Call: list_workspaces
Returns:
[
  {
    "name": "production",
    "display_name": "Production",
    "description": "Customer-facing deploys and jobs",
    "task_count": 12
  },
  {
    "name": "staging",
    "task_count": 8
  }
]
```

### 2. List tasks in a workspace

```
Call: list_tasks(workspace="production")
Returns:
[
  { "name": "deploy-api", "description": "Deploy API server" },
  { "name": "run-tests", "description": "Run test suite" },
  { "name": "backup-db", "description": "Backup production database" }
]
```

### 3. Get task details

```
Call: get_task(workspace="production", task_name="deploy-api")
Returns:
{
  "name": "deploy-api",
  "description": "Deploy API server",
  "input": {
    "version": { "type": "string", "required": true },
    "environment": { "type": "string", "default": "staging" }
  },
  "flow": [
    { "name": "pull-image", "action": "docker-pull", ... },
    { "name": "start-container", "action": "docker-run", "depends_on": ["pull-image"] }
  ]
}
```

### 4. Execute the task

```
Call: execute_task(
  workspace="production",
  task_name="deploy-api",
  input={ "version": "v1.2.3" }
)
Returns:
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
```

`input` must be a JSON object mapping the task's field names to values. If your
MCP client serializes it as a JSON-encoded string (some clients do), the server
parses it back into an object automatically; anything that isn't a JSON object
is rejected with an error rather than silently run with default values.

### 5. Monitor job status

```
Call: get_job_status(job_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890")
Returns:
{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "running",
  "started_at": "2025-03-12T14:30:00Z",
  "steps": [
    { "name": "pull-image", "status": "completed", "duration_secs": 45 },
    { "name": "start-container", "status": "running", "started_at": "2025-03-12T14:30:50Z" }
  ]
}
```

### 6. Retrieve logs

```
Call: get_job_logs(job_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890")
Returns:
[pull-image] Pulling image us-docker.io/stroem/api:v1.2.3...
[pull-image] Pull complete (45.2s)
[start-container] Starting container stroem-api-prod...
[start-container] Container started with ID abc123def456
[start-container] Health check: passed
```

This shows the tail of the log (last 256 KiB by default); pass `tail_bytes` for more history, up to `log_storage.read.tail_max_bytes`.

`get_job_status` masks secret values (`••••••`) in what it returns — chiefly
the steps' `error_message` — the same way as `GET /api/jobs/{id}`, including
the secrets of any [pinned commit](https://fremvaerk.github.io/stroem/guides/git-refs/#secrets-and-redaction)
the job is connected to. When those secrets cannot be loaded at the moment, the tool
returns an error, `redaction set unavailable, retry`, instead of the job;
call it again later. When a pinned commit can never be loaded again, the job
is returned with every content string masked. A job that runs on a git ref
carries its `ref`, in `get_job_status` and in `list_jobs`.

`get_job_logs` returns log lines as the steps printed them; log lines are not
masked.

## Use Cases

### AI-Assisted Deployments

Agents can handle multi-step deployments with approval gates, automated rollbacks on health check failures, and notification of relevant teams.

### Infrastructure Troubleshooting

Agents can query job logs, identify failing steps, and suggest remediation steps based on error messages and logs.

### Automation and CI/CD

External CI/CD systems can use MCP to trigger Strøm tasks, poll job status, and integrate results into larger pipelines.

### Interactive Exploration

Agents can explore available tasks, their input requirements, and past execution patterns to answer operational questions.

## Access Control

When ACL is configured on your Strøm server, MCP tools enforce the same permission rules as the REST API:

- **List tools** (`list_workspaces`, `list_tasks`, `list_jobs`): Filter results to only show resources the user has access to. Tasks with `Deny` permission are hidden. `list_jobs` filters before applying `limit`, so a page holds up to `limit` jobs you may see.
- **Read tools** (`get_task`, `get_job_status`, `get_job_logs`, `list_artifacts`, `get_artifact`): Require `View` or `Run` permission. `Deny` returns "not found". A job that runs on a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) is checked against the folder its own commit declares for its task.
- **Mutation tools** (`execute_task`, `cancel_job`): Require `Run` permission. `View`-only users receive an error.

The `list_tasks` response includes a `can_execute` field indicating whether the user can execute each task (`true` for `Run`, `false` for `View`).

**Admin users bypass ACL over MCP**, exactly as they do in the UI and REST API. Access mirrors the user's actual privileges: an admin sees every workspace, task, and job, while a non-admin is limited to what the ACL grants their email or groups. This holds for **both** authentication methods — API keys and the OAuth flow. If you authenticate as an admin and still see empty lists, confirm you connected as an admin user (see Troubleshooting below).

Jobs created via MCP are tagged with `source_type = "mcp"` and the authenticated user's email as `source_id` for audit trail.

## Best Practices

- **Use API keys for integrations**: Create dedicated API keys for MCP connections rather than sharing user credentials.
- **Set token expiry**: When creating API keys for long-running integrations, consider setting an expiry date for security.
- **Monitor MCP requests**: MCP operations are logged with the authenticated user's email as `source_id` for audit tracking.
- **Validate agent outputs**: When agents execute tasks, ensure their input validation matches your requirements (schemas are available via `get_task`).
- **Use descriptive error messages**: Include clear `error_message` fields in your task actions so agents can provide helpful troubleshooting guidance.

## Troubleshooting

### Connection Refused

Ensure your Strøm server is running and accessible at the configured URL. If behind a proxy or load balancer, verify that the `/mcp` endpoint is not blocked or rewritten.

### Authentication Errors

Verify your API key or JWT token is valid and included in the `Authorization` header. API keys must start with `strm_` prefix. Check the server logs for auth-related errors.

### Tool Not Found

Ensure MCP is enabled in `server-config.yaml` and your server has been restarted. Tools are only available after the server starts with `mcp.enabled: true`.

### Workspace or Task Not Found

Verify the workspace and task names are correct (case-sensitive). Use `list_workspaces` and `list_tasks` to confirm availability.

### Empty Lists / "Not Found" for Everything

If every list tool returns empty and reads/executes report "not found", the authenticated identity has no ACL grants. MCP access mirrors the user's actual privileges, so check: (1) you are connected as an **admin** user (admins bypass ACL), or (2) your email or a group you belong to is granted access in the server's `acl` configuration. Note that the DB `is_admin` flag is what matters — it is honored for both API keys and OAuth-authenticated sessions.

## See Also

- [Workflow Authoring](https://fremvaerk.github.io/stroem/guides/workflow-basics/) — Learn how to define tasks and actions
- [API Reference](https://fremvaerk.github.io/stroem/reference/api/) — REST API documentation
- [Task Actions](https://fremvaerk.github.io/stroem/guides/action-types/) — Available action types for tasks

---

## Workflow YAML Reference

Workflow files live in `.workflows/` and contain `actions`, `tasks`, `triggers`, `connections`, and `secrets`. This page documents every available field.

## Top-Level Structure

```yaml
secrets:
  MY_SECRET: "{{ env.MY_SECRET }}"

connection_types:
  <name>: { ... }

connections:
  <name>: { ... }

actions:
  <name>: { ... }

tasks:
  <name>: { ... }

triggers:
  <name>: { ... }

on_success:
  - action: <name>
on_error:
  - action: <name>
```

All top-level keys are optional. A file typically defines `actions` and `tasks`.

The top-level `on_success` and `on_error` hooks act as workspace-level fallbacks — they fire for top-level jobs when the task itself has no hooks defined for that event type.

---

## Actions

Actions are the smallest execution unit. Each action defines what runs on a worker.

### Common fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | Action type: `script`, `docker`, `pod`, `task`, `agent`, or `approval` |
| `name` | string | — | Human-readable display name |
| `description` | string | — | What this action does |
| `input` | map | `{}` | Input parameter schema (see [Input fields](#input-fields)) |
| `output` | object | — | Output schema (see [Output fields](#output-fields)) |
| `env` | map | — | Environment variables. Values support Tera templates |
| `tags` | list | `[]` | Worker routing tags for step claiming |
| `workdir` | string | — | Working directory override |
| `resources` | object | — | CPU/memory requests (see [Resources](#resources)) |
| `retry` | object | — | Default retry config. Step-level `retry` overrides action-level (see [Retry](#retry)) |

### `type: script`

Runs a script in a [runner](https://fremvaerk.github.io/stroem/guides/runners/) environment with workspace files mounted.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `script` | string | — | Inline script content. Mutually exclusive with `source` |
| `source` | string | — | Path to script file (relative to workspace root). Mutually exclusive with `script` |
| `runner` | string | `local` | Execution environment: `local`, `docker`, or `pod` |
| `language` | string | `shell` | Script language: `shell`, `python`, `javascript` (or `js`), `typescript` (or `ts`), `go` |
| `dependencies` | list | `[]` | Packages to install before running. Requires non-shell language |
| `interpreter` | string | — | Override auto-detected interpreter binary (e.g., `python3.11`) |
| `manifest` | object | — | K8s pod manifest overrides. Only valid with `runner: pod` |

Either `script` or `source` is required. Cannot use `image`, `cmd`, or `entrypoint`.

**Toolchain preferences** (auto-detected when `interpreter` is not set):
- Python: `uv` > `python3` > `python`
- JavaScript: `bun` > `node`
- TypeScript: `bun` > `deno`
- Shell: `bash` > `sh`

```yaml
actions:
  run-tests:
    type: script
    language: python
    dependencies: [pytest, requests]
    script: |
      import pytest
      pytest.main(["-v", "tests/"])
```

### `type: docker`

Runs a container image as-is, without workspace mounting.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `image` | string | **required** | Docker image URI |
| `cmd` | string | — | Shell command to run in the container |
| `entrypoint` | list | — | Container entrypoint override |
| `command` | list | — | Container command args |

Cannot use `runner`, `script`, `source`, `language`, `dependencies`, or `interpreter`.

```yaml
actions:
  run-migration:
    type: docker
    image: myapp/migrations:latest
    command: ["migrate", "--target", "latest"]
```

### `type: pod`

Runs a Kubernetes pod. Same fields as `type: docker`, plus:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `manifest` | object | — | Raw JSON/YAML deep-merged into the generated pod spec |

The `manifest` field supports service accounts, node selectors, tolerations, resource limits, annotations, sidecars, and other pod-level configuration.

```yaml
actions:
  gpu-inference:
    type: pod
    image: myapp/inference:latest
    manifest:
      spec:
        containers:
          - name: main
            resources:
              limits:
                nvidia.com/gpu: "1"
        tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
```

### `type: task`

References another task, creating a child job. The step is dispatched server-side (workers never claim it).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `task` | string | **required** | Name of the task to execute. May be another workspace's task (`ws.task`) |
| `ref` | string | — | Branch, tag or full commit SHA of the task's owner workspace to run the task from — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/). Only valid on `type: task` actions |

Cannot use any other execution fields (`cmd`, `script`, `source`, `image`, `runner`, `language`, `manifest`, etc.).

```yaml
actions:
  run-deploy:
    type: task
    task: deploy-pipeline

tasks:
  orchestrate:
    flow:
      deploy:
        action: run-deploy
        input:
          env: "production"

  deploy-pipeline:
    input:
      env: { type: string }
    flow:
      # ... steps
```

Child jobs have a max nesting depth of 10 levels.

With `ref`, the task is looked up at that ref of its owner workspace and the
child runs pinned to the commit the ref resolved to when the parent job was
created. A task may call itself at another ref (bounded by the nesting
depth). A `type: task` action carrying `ref` cannot be used as a hook.

### `type: agent`

Calls an LLM as a workflow step. The step is dispatched server-side (workers never claim it).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `provider` | string | **required** | Provider ID from server config |
| `prompt` | string | **required** | Tera template for the user message |
| `system_prompt` | string | — | Tera template for system/instruction message |
| `model` | string | — | Override provider's default model |
| `max_tokens` | integer | — | Override provider's max tokens |
| `temperature` | number | — | Override provider's temperature (0–2) |
| `output` | object | — | Output schema (converted to JSON Schema at dispatch) |

Cannot use any other execution fields (`cmd`, `script`, `source`, `image`, `runner`, `language`, `manifest`, `task`, etc.).

`ref` is not supported with agents yet: a flow step that names an agent
action with `ref` is rejected with `400`, and a `tools: [{task: …}]` entry
with `ref` is a validation error (a tool call naming it is rejected). Agent
steps *inside* a job that runs on a [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) work and use
that commit's definitions.

```yaml
actions:
  classify-ticket:
    type: agent
    provider: anthropic-main
    system_prompt: "You are a support ticket classifier."
    prompt: "Classify this ticket: {{ input.ticket_body }}"
    output:
      category:
        type: string
        required: true
        options: [bug, feature_request, question]
      confidence:
        type: number

tasks:
  handle-support:
    input:
      ticket_body: { type: string }
    flow:
      classify:
        action: classify-ticket
        input:
          ticket_body: "{{ input.ticket_body }}"
      route-bug:
        action: escalate
        depends_on: [classify]
        when: "{{ classify.output.category == 'bug' }}"
```

### `type: approval`

Pauses execution for human approval/rejection. The step is dispatched server-side (workers never claim it).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `message` | string | — | Tera template for the approval message shown to approver |

Cannot use any other execution fields (`cmd`, `script`, `source`, `image`, `runner`, `language`, `manifest`, `task`, `provider`, `prompt`, etc.).

```yaml
actions:
  approve-deployment:
    type: approval
    message: |
      Deploy to production?
      Version: {{ input.version }}
      Environment: {{ input.env }}

tasks:
  deploy-workflow:
    input:
      version: { type: string }
      env: { type: string }
    flow:
      manual-approval:
        action: approve-deployment
        input:
          version: "{{ input.version }}"
          env: "{{ input.env }}"
      deploy:
        action: run-deployment
        depends_on: [manual-approval]
        input:
          version: "{{ input.version }}"
```

When approved, the step's output is `{ "approved": true, "approved_by": "user@example.com", "approved_at": "2025-03-21T10:30:00Z", "input": {...} }`. When rejected, the step fails and downstream steps are skipped.

---

## Input Fields

Input parameters define the schema for action and task inputs.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | `string`, `text`, `integer`, `number`, `boolean`, `date`, `datetime`, `json`, `file`, or a connection type name. Alias: `bool` is accepted as a shorthand for `boolean` and normalized on load |
| `name` | string | — | Human-readable label for the UI |
| `description` | string | — | Help text |
| `required` | bool | `false` | Whether the field must be provided |
| `secret` | bool | `false` | Mask value in UI and logs |
| `default` | any | — | Default value when not provided |
| `options` | list | — | Predefined dropdown choices |
| `allow_custom` | bool | `false` | Allow values outside `options` list |
| `multiple` | bool | `false` | Accept multiple values (requires `options`). Submitted value is a JSON array; see [Multi-select](https://fremvaerk.github.io/stroem/guides/input-and-output/#multi-select) |
| `order` | integer | — | Display order in UI (lower = earlier). Fields without `order` appear last |

```yaml
input:
  environment:
    type: string
    description: Target environment
    required: true
    options: [staging, production]
  version:
    type: string
    default: "latest"
  dry_run:
    type: boolean
    default: false
    order: 99
```

A `file` field (task or action level) takes an uploaded file; `{{ input.<field> }}` renders to a read-only path. It cannot have `default`, `options`, `allow_custom`, `multiple` or `secret`. `file` is a reserved name: a connection type cannot be called `file`. See [File Inputs](https://fremvaerk.github.io/stroem/guides/file-inputs/).

When `type` is not a primitive, it references a [connection type](#connection-types). The input value (a connection name) is resolved to the full connection object at execution time.

---

## Output Fields

Output schemas describe the structured output of an action. Script actions emit output by printing `OUTPUT: {json}` to stdout. Agent actions use `output` to instruct the LLM to respond with structured JSON.

```yaml
actions:
  build:
    type: script
    script: |
      echo "Building..."
      echo 'OUTPUT: {"artifact": "dist/app.tar.gz", "size": 1024}'
    output:
      artifact: { type: string }
      size: { type: integer }
```

Output fields are defined directly under the `output:` key as a flat map of field names to field definitions.

**Output field properties:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | String | Required | `string`, `integer`, `number`, `boolean`, `array`, `object`. Alias: `bool` is accepted for `boolean` |
| `description` | String | — | Human-readable description |
| `required` | Boolean | `false` | Whether the field must be present |
| `default` | Any | — | Default value |
| `options` | Array | — | Allowed values (maps to JSON Schema `enum`) |

For agent actions, `output` is converted to JSON Schema at dispatch time and injected into the system prompt.

---

## Resources

CPU and memory requests for runner containers.

| Field | Type | Description |
|-------|------|-------------|
| `cpu` | string | CPU request (e.g., `"500m"`, `"1"`) |
| `memory` | string | Memory request (e.g., `"256Mi"`, `"1Gi"`) |

```yaml
actions:
  heavy-compute:
    type: script
    runner: pod
    resources:
      cpu: "2"
      memory: "4Gi"
    script: "..."
```

---

## Tasks

Tasks compose actions into a DAG of steps.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | string | — | Human-readable display name |
| `description` | string | — | What this task does |
| `mode` | string | `distributed` | Execution mode: `distributed` |
| `folder` | string | — | UI folder path (e.g., `deploy/staging`). Use `/` for nesting |
| `input` | map | `{}` | Input parameter schema (see [Input fields](#input-fields)) |
| `flow` | map | **required** | Steps — map of step name to [flow step](#flow-steps) |
| `timeout` | duration | — | Job-level timeout. Max `7d` (604800s). Cancels entire job when exceeded |
| `retry` | object | — | Retry entire task on failure, creating a new job (see [Retry](#retry)) |
| `on_success` | list | `[]` | [Hooks](#hooks) to run when the job succeeds |
| `on_error` | list | `[]` | [Hooks](#hooks) to run when the job fails |

```yaml
tasks:
  deploy-pipeline:
    name: Deploy Pipeline
    description: Build, deploy, and verify
    folder: deploy
    timeout: 30m
    input:
      env: { type: string, default: "staging" }
    flow:
      build:
        action: build-app
      deploy:
        action: deploy
        depends_on: [build]
    on_error:
      - action: notify
        input:
          message: "Deploy failed: {{ hook.error_message }}"
```

---

## Flow Steps

Each entry in a task's `flow` map defines a step. Steps can reference a named action or define one inline.

### Step fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `action` | string | **required** | Action name to execute. Omit when using inline action |
| `ref` | string | — | Run `action` from this branch, tag or full commit SHA of its owner workspace — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/). The action need not exist on the default branch. Not allowed when the action is `type: agent` |
| `name` | string | — | Human-readable step display name |
| `description` | string | — | What this step does |
| `depends_on` | list | `[]` | Dependencies that must reach a terminal state before this one is decided (see [Dependencies](#dependencies) for the full `{step, accept}` / `all` / `any` shape) |
| `input` | map | `{}` | Input values passed to the action. Values support Tera templates |
| `continue_on_failure` | bool | `false` | Self-scoped only: if this step fails, its failure does not fail the job. Has no effect on whether anything downstream runs — that's `depends_on`'s `accept` sets on each dependent (see [Dependencies](#dependencies)) |
| `timeout` | duration | — | Step execution timeout. Max `24h` (86400s) |
| `when` | string | — | Tera condition. Falsy values: empty string, `"false"`, `"null"`, `"none"` (case-insensitive), `"[]"`, `"{}"` and any number equal to zero (`"0"`, `"0.0"`) |
| `for_each` | string or list | — | Tera expression or literal JSON array. Creates one instance per item |
| `sequential` | bool | `false` | Run `for_each` instances one at a time instead of in parallel |
| `retry` | object | — | Retry configuration for this step (see [Retry](#retry)) |

### Inline actions

Instead of referencing a named action, define one inline by adding action fields directly on the step:

```yaml
flow:
  greet:
    type: script
    script: "echo Hello {{ input.name }}"
    input:
      name: "{{ input.name }}"
```

All [action fields](#actions) are valid on inline steps (e.g., `type`, `script`, `source`, `image`, `runner`, `language`, `env`, `tags`).

### Dependencies

```yaml
flow:
  a:
    action: step-a
  b:
    action: step-b
  c:
    action: step-c
    depends_on: [a, b]    # waits for both a and b to complete
```

Steps without `depends_on` start immediately. Each entry in a `depends_on` list is one of four shapes; the list itself is an implicit `all` group:

| Shape | Meaning |
|---|---|
| `plain-name` | Sugar for `{step: plain-name, accept: [completed]}` — today's default: the dependency must complete |
| `{step: name, accept: [...]}` | Satisfied iff `name`'s outcome is in the `accept` list |
| `{step: name, accept: any}` | Sugar for all five outcomes below — "wait for it, don't care how it ends" |
| `{all: [...]}` | Satisfied iff every nested entry is satisfied (an explicit, nestable AND) |
| `{any: [...]}` | Satisfied iff at least one nested entry is satisfied (OR) |

`accept` is a non-empty list drawn from five **outcomes**, or the literal string `any` (any outcome — not to be confused with the `any:` group, which needs any one of its entries satisfied):

| Outcome | Meaning |
|---|---|
| `completed` | The dependency's status is `completed` |
| `failed` | The dependency's status is `failed` |
| `cancelled` | The dependency's status is `cancelled` |
| `skipped` | The dependency was skipped by its **own** choice — its own `when` was false, or its own `for_each` was empty |
| `omitted` | The dependency was itself blocked by **its own** `depends_on` tree — not its own choice |

**Readiness is uniform**: the gate waits until every step named anywhere in the tree — including nested inside `all`/`any` — has reached a terminal state, then evaluates the whole tree exactly once. This holds even when the tree is already conclusively satisfied or unsatisfiable by one child — there's no fail-fast, and no short-circuiting an `any` ahead of a still-running sibling. A step whose tree is not satisfied is skipped with `skip_reason: unreachable`.

`all`/`any` nest arbitrarily (an `any` may contain an `all`, a `step` entry, or a bare name). The same step name may appear more than once across *different* branches with different `accept` sets — that's a legitimate way to express an OR over different outcomes of the same dependency, and is accepted. It's rejected only when the same name is a **direct sibling** of the same `all`/`any` node (including the implicit top-level group): `depends_on: [a, a]`-style duplicates must be deduplicated.

`continue_on_failure` is unrelated to any of this — it only controls whether a step's own failure fails the *job* (see the `continue_on_failure` row above), never whether anything downstream runs. See the [Conditionals guide](https://fremvaerk.github.io/stroem/guides/conditionals/) for worked examples, and the [0.18 upgrade guide](https://fremvaerk.github.io/stroem/operations/upgrade-0-18-dependency-conditions/) if migrating from `continue_on_failure`/`continue_when_skipped`-based gating.

Every skipped step records a `skip_reason` (`condition`, `empty`, or `unreachable`; a legacy `cascade` reason can still appear on rows written before 0.18.0 and is read identically to `unreachable`); see the [Conditionals guide](https://fremvaerk.github.io/stroem/guides/conditionals/#outcomes-and-skip-reasons).

### Conditional steps (`when`)

```yaml
flow:
  deploy:
    action: deploy-app
    when: "{{ input.env == 'production' }}"
    input:
      env: "{{ input.env }}"
```

Evaluated before `for_each`. If the condition errors, the step fails (not silently skipped). See the [Conditionals guide](https://fremvaerk.github.io/stroem/guides/conditionals/) for details.

### For-each loops

```yaml
flow:
  deploy:
    action: deploy-to-region
    for_each: ["us-east-1", "eu-west-1", "ap-south-1"]
    input:
      region: "{{ each.item }}"
      index: "{{ each.index }}"
```

Creates `deploy[0]`, `deploy[1]`, `deploy[2]`. Inside instances, `each.item` is the current value and `each.index` is the zero-based position.

Downstream steps receive the aggregated output as an array via `{{ step_name.output }}`.

**Limits**: max 10,000 items. Step names must not contain `[` or `]`. See the [Loops guide](https://fremvaerk.github.io/stroem/guides/loops/) for sequential mode, error handling, and sub-job fan-out.

### Timeouts

Accepts human-readable durations: `30s`, `5m`, `1h30m`, or a plain integer (seconds).

```yaml
flow:
  build:
    action: build-app
    timeout: 10m
```

Enforced both server-side (recovery sweeper) and worker-side (process cancellation).

---

## Retry

Strøm supports two layers of retry: **step-level retry** (retries the step in-place with backoff) and **task-level retry** (retries the entire task by creating a new job).

### Retry configuration fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `max_attempts` | integer | `3` | Maximum number of attempts (including first attempt). Range: 1–10 |
| `delay` | duration | `"30s"` | Delay before first retry. Max `1h` (3600s). Accepts `30s`, `5m`, `1h`, or integer seconds |
| `backoff` | string | `"fixed"` | Strategy: `"fixed"` (constant delay) or `"exponential"` (delay doubles each attempt, capped at 64x) |
| `jitter` | bool | `false` | Add 0–25% random jitter to delay (reduces thundering herd on mass retries) |

### Step-level retry

Retries a single step in-place. When a step fails, the server waits for the configured delay, then resets the step and allows a worker to claim and re-execute it.

```yaml
actions:
  fetch-data:
    type: script
    script: |
      curl https://api.example.com/data || exit 1
      echo 'OUTPUT: {"result": "ok"}'

tasks:
  main:
    flow:
      get-data:
        action: fetch-data
        retry:
          max_attempts: 3
          delay: "10s"
          backoff: exponential
```

**How it works:**
- Attempt 1 fails
- Wait 10s, then retry (Attempt 2)
- If Attempt 2 fails, wait 20s, then retry (Attempt 3)
- If Attempt 3 fails, the step fails; its dependents run only if their own `depends_on` edge to this step accepts `failed` — otherwise they're skipped `unreachable`
- All data from previous attempts is discarded; output is only captured from the final attempt

**Step-level vs. action-level:** If both action and step define retry, the step's configuration takes precedence:

```yaml
actions:
  api-call:
    type: script
    script: "curl https://api.example.com"
    retry:
      max_attempts: 2     # Action default
      delay: "5s"

tasks:
  main:
    flow:
      call:
        action: api-call
        retry:
          max_attempts: 4 # Step overrides action's max_attempts
          delay: "10s"
```

### Action-level retry defaults

Actions can define default retry behavior that applies to all steps using that action, unless the step overrides it:

```yaml
actions:
  unstable-task:
    type: script
    script: "some-flaky-operation"
    retry:
      max_attempts: 3
      delay: "30s"
      backoff: exponential

tasks:
  main:
    flow:
      step1:
        action: unstable-task
        # Uses action's retry config: 3 max_attempts, 30s delay, exponential backoff

      step2:
        action: unstable-task
        retry:
          max_attempts: 5 # Overrides action default
```

### Task-level retry

Retries the entire task by creating a new job when the original job fails. This is useful when you want to retry all steps from the beginning on failure.

```yaml
tasks:
  deploy:
    retry:
      max_attempts: 2
      delay: "60s"
    flow:
      build:
        action: build-app
      test:
        action: run-tests
      deploy:
        action: deploy-app
```

**How it works:**
- Original job fails at `deploy` step
- Server waits 60s, then creates a **new job** with the same task and input
- New job starts from `build` step
- If the new job also fails, no further retries (max_attempts: 2 exhausted)
- On final failure, `on_error` hooks fire only once (not after each retry attempt)

**Retry chain:** every retry job's `retry_of_job_id` points at the first job of the chain; each failed attempt's `retry_job_id` points at the attempt that replaced it:

```
Job 1 (failed)                retry_job_id → 2
  └─ Job 2 (failed)           retry_of_job_id → 1, retry_job_id → 3
      └─ Job 3 (success)      retry_of_job_id → 1
```
If the first job has since been deleted by [retention](https://fremvaerk.github.io/stroem/operations/retention/), the pointer is cleared (`ON DELETE SET NULL`), and the next retry points at the attempt that just failed instead — not at the earliest attempt that still exists.

**Task retry vs. step retry:**
- **Step retry**: Faster, smaller scope, good for transient failures (network timeouts, temporary service degradation)
- **Task retry**: Slower, full job restart, good for cascading failures (dependency issues, state cleanup needed)

### Backoff strategies

#### Fixed backoff

Constant delay between retries. Simple and predictable.

```yaml
retry:
  delay: "30s"
  backoff: fixed      # or omit (default)
```

Delays: 30s, 30s, 30s, ...

#### Exponential backoff

Delay doubles with each attempt (up to a maximum of 64x the initial delay).

```yaml
retry:
  delay: "10s"
  backoff: exponential
  jitter: true        # Recommended for high concurrency
```

Delays without jitter: 10s, 20s, 40s, 80s (capped at 640s), 640s, ...

With jitter, each delay gets ±0–25% random variance:
- 10s → 7.5s–12.5s
- 20s → 15s–25s
- 40s → 30s–50s

Jitter reduces synchronized retries across many workers (thundering herd problem).

### Limits and constraints

| Constraint | Limit | Example |
|-----------|-------|---------|
| `max_attempts` | 1–10 | `max_attempts: 5` (5 total attempts) |
| `delay` | 0–3600s (1h) | `delay: "30s"` or `delay: "1h"` |
| Exponential cap | 64x initial delay | With `delay: 10s`, max is 640s |
| Step timeout vs. retry | Step timeout applies per attempt | Step with `timeout: 5m` and retry will try up to 5m per attempt |

### Interactions

#### Retry with `continue_on_failure`

Step retry exhausts first; `continue_on_failure` then decides whether the job still counts as succeeded, independent of whether `next` runs (which is `next`'s own `accept` set):

```yaml
flow:
  unstable:
    action: flaky-operation
    retry:
      max_attempts: 3
    continue_on_failure: true   # job still completes even if all retries fail

  next:
    action: next-step
    depends_on:
      - step: unstable
        accept: [completed, failed]
    # Runs even if all 3 retry attempts fail
```

#### Retry with `for_each` (loop instances)

Loop instances inherit the step's retry config:

```yaml
flow:
  process:
    action: process-item
    for_each: [a, b, c]
    retry:
      max_attempts: 2
    # Each instance (process[0], process[1], process[2]) gets max 2 attempts
```

#### Retry with `when` (conditional steps)

Retry applies only if the step actually runs:

```yaml
flow:
  deploy:
    action: deploy-app
    when: "{{ input.env == 'prod' }}"
    retry:
      max_attempts: 3
    # Retries only if condition is true; skipped if condition is false
```

#### Retry with timeouts

Step timeout applies per attempt. Task timeout applies to the entire retry chain:

```yaml
tasks:
  main:
    timeout: 10m    # Total time across all retry attempts
    flow:
      fetch:
        action: fetch-data
        timeout: 2m  # 2m per attempt
        retry:
          max_attempts: 3
          delay: "30s"
```

If attempts are 2m, 2m, 2m (6m) plus delays (30s, 30s = 1m), total is 7m — well under the 10m task timeout.

#### Retry with hooks

Hooks fire after all retry attempts are exhausted:

```yaml
tasks:
  main:
    flow:
      unstable:
        action: flaky-operation
        retry:
          max_attempts: 3
    on_error:
      - action: notify-ops
        # Fires only after all 3 attempts fail, not after each attempt
```

### Viewing retry history

Retried steps are visible in the job detail UI with:
- Retry badge showing attempt number (e.g., "Attempt 2 of 3")
- Timestamp of each attempt
- Previous attempt's output (if available)
- Backoff delay until next retry

Task-level retries create new jobs that appear in the job list, linked by `retry_of_job_id`.

---

## Hooks

Hooks fire when a job reaches a terminal state. Defined on tasks or at the top level of the workflow file.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `action` | string | **required** | Action to execute. Can be a `type: task` action for full child job |
| `input` | map | `{}` | Input values. Supports Tera templates with `hook` context |

`ref` is not supported on hooks yet: a hook with `ref`, or one whose action
is a `type: task` action carrying `ref`, is a validation error and is never
fired.

### Hook template context

| Variable | Type | Description |
|----------|------|-------------|
| `hook.workspace` | string | Workspace name |
| `hook.task_name` | string | Task that completed/failed |
| `hook.job_id` | string | Job UUID |
| `hook.status` | string | `"completed"` or `"failed"` |
| `hook.is_success` | bool | `true` if completed |
| `hook.error_message` | string/null | All failed step errors combined |
| `hook.source_type` | string | Original job source (`"api"`, `"trigger"`, etc.) |
| `hook.source_id` | string/null | Original job source ID |
| `hook.started_at` | string/null | ISO 8601 timestamp |
| `hook.completed_at` | string/null | ISO 8601 timestamp |
| `hook.duration_secs` | number/null | Execution duration in seconds |
| `hook.failed_steps` | list | Failed step details (see below) |
| `hook.revision` | string/null | Workspace revision pinned on the job (git SHA or folder hash) |
| `hook.ref` | string/null | The [git ref](https://fremvaerk.github.io/stroem/guides/git-refs/) the job runs at; `null` unless it is a pinned job |

Each entry in `hook.failed_steps`:

| Field | Type | Description |
|-------|------|-------------|
| `step_name` | string | Name of the failed step |
| `action_name` | string | Action that was executed |
| `error_message` | string/null | The step's error message |
| `continue_on_failure` | bool | Whether this row's own flow step has `continue_on_failure` set. A loop instance row reports its placeholder's flag |
| `tolerated` | bool | Same value as `continue_on_failure` above — this failure does not by itself fail the job; other untolerated failures or a cancellation can still decide the outcome |

```yaml
tasks:
  deploy:
    flow:
      # ...
    on_success:
      - action: notify
        input:
          message: "Deploy completed in {{ hook.duration_secs }}s"
    on_error:
      - action: notify
        input:
          message: "Deploy failed: {{ hook.error_message }}"
```

Hook jobs use `source_type = "hook"` and never trigger further hooks (recursion guard).

---

## Triggers

### Scheduler (`type: scheduler`)

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | `scheduler` |
| `cron` | string | **required** | Cron expression (5-field, or 6-field with optional seconds) |
| `task` | string | **required** | Task to execute. May be another workspace's task (`ws.task`) |
| `ref` | string | — | Branch, tag or full commit SHA of the task's owner workspace — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/) |
| `input` | map | `{}` | Input values |
| `enabled` | bool | `true` | Whether the trigger is active |
| `timezone` | string | UTC | IANA timezone name (e.g., `Europe/Copenhagen`) |
| `concurrency` | string | `allow` | `allow`, `skip`, or `cancel_previous` |

```yaml
triggers:
  nightly-backup:
    type: scheduler
    cron: "0 2 * * *"
    task: run-backup
    timezone: "Europe/Copenhagen"
    concurrency: skip
```

### Webhook (`type: webhook`)

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | `webhook` |
| `name` | string | **required** | URL-safe name (alphanumeric, `-`, `_`). Endpoint: `/hooks/{name}` |
| `task` | string | **required** | Task to execute. May be another workspace's task (`ws.task`) |
| `ref` | string | — | Branch, tag or full commit SHA of the task's owner workspace — see [Git Refs](https://fremvaerk.github.io/stroem/guides/git-refs/) |
| `secret` | string | — | Auth via `?secret=xxx` or `Authorization: Bearer xxx` |
| `input` | map | `{}` | Default input values. Request body/headers/query merged in |
| `enabled` | bool | `true` | Whether the webhook is active |
| `mode` | string | `async` | `async` (fire-and-forget) or `sync` (wait for job completion) |
| `timeout_secs` | integer | `30` | Max wait in sync mode (1–300). Returns 202 on timeout |

```yaml
triggers:
  github-push:
    type: webhook
    name: github-ci
    task: ci-pipeline
    secret: "whsec_your_secret"
    mode: sync
    timeout_secs: 120
```

### Event Source (`type: event_source`)

Long-running queue consumer processes that create jobs via `OUTPUT: ` prefix protocol.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | `event_source` |
| `task` | string | **required** | Consumer task that runs continuously |
| `target_task` | string | **required** | Task to create jobs for from emitted events |
| `env` | map | `{}` | Tera-templated environment variables for consumer execution |
| `input` | map | `{}` | Default input merged into each emitted job |
| `restart_policy` | string | `always` | `always`, `on_failure`, or `never` |
| `backoff_secs` | integer | `5` | Initial restart delay with exponential backoff |
| `max_in_flight` | integer | — | Max concurrent pending/running target jobs (optional, unlimited if omitted) |
| `enabled` | bool | `true` | Whether the consumer is active |

```yaml
triggers:
  sqs-events:
    type: event_source
    task: sqs-consumer
    target_task: process-order
    env:
      QUEUE_URL: "{{ secret.sqs_url }}"
    input:
      priority: high
    restart_policy: always
    backoff_secs: 5
    max_in_flight: 10
```

The consumer task runs as a normal job. Any step's `OUTPUT: ` lines (followed by valid JSON) create jobs for `target_task`. The `input` defaults are merged with the emitted JSON (emitted data wins on conflict).

`ref` is not supported on event-source triggers yet: it is a validation
error, and the server does not start such a consumer.

---

## Connection Types

Define schemas for reusable connection configurations.

| Field | Type | Description |
|-------|------|-------------|
| `properties` | map | Map of property names to property definitions |

### Property fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | **required** | `string`, `text`, `integer`, `number`, `boolean`, `date`, or `datetime`. Alias: `bool` is accepted for `boolean` |
| `required` | bool | `false` | Whether the property must be provided |
| `default` | any | — | Default value |
| `secret` | bool | `false` | Mask in UI and logs |

```yaml
connection_types:
  postgres:
    host: { type: string, required: true }
    port: { type: integer, default: 5432 }
    database: { type: string, required: true }
    username: { type: string, required: true }
    password: { type: string, required: true, secret: true }
```

## Connections

Named instances of connection types.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | string | — | Connection type reference (for validation). Optional for untyped connections |
| *(other keys)* | any | — | Property values (flattened alongside `type`) |

```yaml
connections:
  prod-db:
    type: postgres
    host: "db.example.com"
    port: 5432
    database: myapp
    username: "{{ secrets.DB_USER }}"
    password: "{{ secrets.DB_PASS }}"
```

Use connection names as input values when the input field's `type` references a connection type. The name is resolved to the full connection object at execution time.

---

## Secrets

Key-value map rendered through Tera at workspace load time. Typically sources values from environment variables:

```yaml
secrets:
  DB_PASSWORD: "{{ env.DB_PASSWORD }}"
  API_KEY: "{{ env.MY_API_KEY }}"
  STATIC_VALUE: "hardcoded-is-fine-too"
```

Secrets are available in connection values and action templates via `{{ secrets.DB_PASSWORD }}`.

---

## Duration Format

Timeout fields accept human-readable durations or plain integers (seconds):

| Format | Example | Seconds |
|--------|---------|---------|
| Seconds | `30s` | 30 |
| Minutes | `5m` | 300 |
| Hours | `1h` | 3600 |
| Combined | `1h30m` | 5400 |
| Plain integer | `300` | 300 |

---

## Hello World

This example creates a two-step workflow that greets someone and then shouts the greeting in uppercase.

## Workflow file

Create `workspace/.workflows/hello.yaml`:

```yaml
actions:
  greet:
    type: script
    script: "echo Hello {{ input.name }} && echo 'OUTPUT: {\"greeting\": \"Hello {{ input.name }}\"}'"
    input:
      name: { type: string, required: true }

  shout:
    type: script
    script: "echo {{ input.message }} | tr '[:lower:]' '[:upper:]'"
    input:
      message: { type: string, required: true }

tasks:
  hello-world:
    mode: distributed
    input:
      name: { type: string, default: "World" }
    flow:
      say-hello:
        action: greet
        input:
          name: "{{ input.name }}"
      shout-it:
        action: shout
        depends_on: [say-hello]
        input:
          message: "{{ say_hello.output.greeting }}"
```

## What this does

1. **`greet` action** prints "Hello World" and emits structured output with `OUTPUT: {json}`
2. **`shout` action** takes a message and converts it to uppercase
3. **`hello-world` task** runs `say-hello` first, then passes its output to `shout-it`

The two steps form a dependency chain: `say-hello` → `shout-it`.

## Key concepts demonstrated

- **Actions** define reusable commands with typed input parameters
- **Tasks** compose actions into a DAG via `depends_on`
- **Structured output** uses `OUTPUT: {json}` prefix in stdout
- **Data passing** uses Tera templates: `{{ say_hello.output.greeting }}`
- **Step name sanitization**: `say-hello` becomes `say_hello` in templates (hyphens → underscores)

## Running it

```bash
# Via API
curl -s -X POST http://localhost:8080/api/workspaces/default/tasks/hello-world/execute \
  -H "Content-Type: application/json" \
  -d '{"input": {"name": "World"}}' | jq .

# Via CLI
stroem trigger hello-world --input '{"name": "World"}'
```

## Expected output

Step `say-hello`:
```
Hello World
OUTPUT: {"greeting": "Hello World"}
```

Step `shout-it`:
```
HELLO WORLD
```

---

## Deploy Pipeline

This example creates a deployment pipeline with a health check, deployment step, and a notification.

## Workflow file

Create `workspace/.workflows/deploy.yaml`:

```yaml
actions:
  check-status:
    type: script
    script: "echo 'Checking system status...' && sleep 1 && echo 'OUTPUT: {\"healthy\": true}'"

  deploy:
    type: script
    source: actions/deploy.sh
    input:
      env: { type: string, default: "staging" }

  notify:
    type: script
    script: "echo 'Notification: Deployment to {{ input.env }} completed with status={{ input.status }}'"
    input:
      env: { type: string }
      status: { type: string }

tasks:
  deploy-pipeline:
    mode: distributed
    input:
      env: { type: string, default: "staging" }
    flow:
      health-check:
        action: check-status
      deploy-app:
        action: deploy
        depends_on: [health-check]
        input:
          env: "{{ input.env }}"
      send-notification:
        action: notify
        depends_on: [deploy-app]
        input:
          env: "{{ input.env }}"
          status: "success"
```

## Deploy script

Create `workspace/actions/deploy.sh`:

```bash
#!/bin/bash
set -euo pipefail

echo "Deploying to environment: ${DEPLOY_ENV:-staging}"
echo "Version: ${VERSION:-latest}"
sleep 1
echo "Deployment complete!"
echo "OUTPUT: {\"status\": \"deployed\", \"env\": \"${DEPLOY_ENV:-staging}\"}"
```

## What this does

1. **`health-check`** verifies the system is healthy before deploying
2. **`deploy-app`** runs the deployment script (depends on health check)
3. **`send-notification`** sends a notification after deployment (depends on deploy)

The steps execute sequentially: `health-check` → `deploy-app` → `send-notification`.

## Key concepts demonstrated

- **Script files** referenced via `source:` (relative to workspace root)
- **Linear dependencies** create a sequential pipeline
- **Input propagation** passes task input down to step actions
- **Multiple actions** in a single workflow file

## Running it

```bash
# Deploy to staging (default)
stroem trigger deploy-pipeline

# Deploy to production
stroem trigger deploy-pipeline --input '{"env": "production"}'
```

## Multi-region deploy with `for_each`

Fan out the deployment across multiple regions in parallel using [`for_each`](https://fremvaerk.github.io/stroem/guides/loops/):

```yaml
actions:
  check-status:
    type: script
    script: "echo 'Checking system status...' && echo 'OUTPUT: {\"healthy\": true}'"

  deploy:
    type: script
    script: |
      echo "Deploying to {{ input.region }} ({{ input.env }})..."
      sleep 2
      echo "OUTPUT: {\"region\": \"{{ input.region }}\", \"status\": \"deployed\"}"
    input:
      env: { type: string }
      region: { type: string }

  notify:
    type: script
    script: "echo 'Deployed to {{ input.regions | length }} regions: {{ input.status }}'"
    input:
      regions: { type: string }
      status: { type: string }

tasks:
  multi-region-deploy:
    mode: distributed
    input:
      env: { type: string, default: "staging" }
      regions: { type: string, default: '["us-east-1", "eu-west-1", "ap-south-1"]' }
    flow:
      health-check:
        action: check-status
      deploy-region:
        action: deploy
        depends_on: [health-check]
        for_each: "{{ input.regions }}"
        input:
          env: "{{ input.env }}"
          region: "{{ each.item }}"
      send-notification:
        action: notify
        depends_on: [deploy-region]
        input:
          regions: "{{ deploy_region.output }}"
          status: "success"
```

This creates `deploy-region[0]`, `deploy-region[1]`, `deploy-region[2]` — all running in parallel after the health check passes. The notification step waits for all regions to complete and receives the aggregated output array.

To deploy regions one at a time (e.g., canary rollout), add `sequential: true`:

```yaml
      deploy-region:
        action: deploy
        depends_on: [health-check]
        for_each: "{{ input.regions }}"
        sequential: true
        input:
          env: "{{ input.env }}"
          region: "{{ each.item }}"
```

### Running it

```bash
# Deploy to default regions
stroem trigger multi-region-deploy

# Deploy to specific regions in production
stroem trigger multi-region-deploy \
  --input '{"env": "production", "regions": "[\"us-east-1\", \"eu-west-1\"]"}'
```

## Sub-job fan-out with `type: task` + `for_each`

When each iteration needs multiple steps, use a [`type: task`](https://fremvaerk.github.io/stroem/guides/workflow-basics/#tasks) action so each loop item spawns a full child job with its own steps and logs:

```yaml
actions:
  check-region:
    type: script
    script: |
      echo "Health-checking {{ input.region }}..."
      echo 'OUTPUT: {"healthy": true}'
    input:
      region: { type: string }

  deploy-to-region:
    type: script
    script: |
      echo "Deploying {{ input.env }} to {{ input.region }}..."
      sleep 2
      echo 'OUTPUT: {"status": "deployed"}'
    input:
      env: { type: string }
      region: { type: string }

  smoke-test:
    type: script
    script: |
      echo "Running smoke tests in {{ input.region }}..."
      echo 'OUTPUT: {"passed": true}'
    input:
      region: { type: string }

  # Task action — references the region-deploy task below
  run-region-deploy:
    type: task
    task: region-deploy

tasks:
  # Parent task: fans out across regions
  deploy-all-regions:
    mode: distributed
    input:
      env: { type: string, default: "staging" }
      regions: { type: string, default: '["us-east-1", "eu-west-1", "ap-south-1"]' }
    flow:
      deploy:
        action: run-region-deploy
        for_each: "{{ input.regions }}"
        input:
          env: "{{ input.env }}"
          region: "{{ each.item }}"

  # Child task: multi-step deploy for a single region
  region-deploy:
    mode: distributed
    input:
      env: { type: string }
      region: { type: string }
    flow:
      health-check:
        action: check-region
        input:
          region: "{{ input.region }}"
      deploy:
        action: deploy-to-region
        depends_on: [health-check]
        input:
          env: "{{ input.env }}"
          region: "{{ input.region }}"
      verify:
        action: smoke-test
        depends_on: [deploy]
        input:
          region: "{{ input.region }}"
```

Each region gets its own child job (`region-deploy`) with three steps: health check → deploy → smoke test. All three child jobs run in parallel. The parent `deploy` step fans in when all child jobs complete.

This pattern is useful when each iteration involves complex multi-step logic, or when you want separate log streams and status tracking per region in the UI.

## Adding hooks

Enhance the pipeline with success/failure notifications:

```yaml
tasks:
  deploy-pipeline:
    flow:
      health-check:
        action: check-status
      deploy-app:
        action: deploy
        depends_on: [health-check]
        input:
          env: "{{ input.env }}"
      send-notification:
        action: notify
        depends_on: [deploy-app]
        input:
          env: "{{ input.env }}"
          status: "success"
    on_error:
      - action: notify
        input:
          env: "{{ hook.workspace }}"
          status: "FAILED: {{ hook.error_message }}"
```

---

## CI Pipeline

This example creates a CI pipeline triggered by GitHub webhooks with parallel lint and test steps.

## Workflow file

Create `workspace/.workflows/ci.yaml`:

```yaml
actions:
  lint:
    type: script
    runner: docker
    script: |
      cd /workspace
      npm ci
      npm run lint

  test:
    type: script
    runner: docker
    script: |
      cd /workspace
      npm ci
      npm test

  build:
    type: script
    runner: docker
    script: |
      cd /workspace
      npm ci
      npm run build
      echo "OUTPUT: {\"artifact\": \"dist/app.tar.gz\"}"

  notify-ci:
    type: script
    script: |
      echo "CI result: {{ input.status }} for job {{ input.job_id }}"
      # In production, call Slack/Teams/GitHub API here

    input:
      status: { type: string }
      job_id: { type: string }

tasks:
  ci-pipeline:
    mode: distributed
    input:
      body: { type: string }
    flow:
      lint:
        action: lint
      test:
        action: test
      # lint and test run in parallel (no mutual dependencies)
      build:
        action: build
        depends_on: [lint, test]
    on_success:
      - action: notify-ci
        input:
          status: "success"
          job_id: "{{ hook.job_id }}"
    on_error:
      - action: notify-ci
        input:
          status: "failed: {{ hook.error_message }}"
          job_id: "{{ hook.job_id }}"

triggers:
  github-push:
    type: webhook
    name: github-ci
    task: ci-pipeline
    secret: "whsec_your_secret_here"
    enabled: true
```

## What this does

1. **`lint`** and **`test`** run in parallel (no dependency between them)
2. **`build`** waits for both lint and test to complete
3. An `on_success` hook notifies when the whole pipeline passes; an `on_error` hook notifies when any step fails — a hook always fires exactly once on the real terminal outcome, which a flow step reaching past a failure (via its own `accept` set) can't guarantee (see [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/))

The DAG looks like:

```
lint  ──┐
        ├──> build
test  ──┘
```

## Key concepts demonstrated

- **Parallel steps**: `lint` and `test` have no mutual dependencies, so they run concurrently
- **Docker runner**: `runner: docker` runs steps inside containers with workspace at `/workspace`
- **Hooks, not a dependent step**: notifying after success or failure is an `on_success` / `on_error` hook, not a flow step. `continue_on_failure` is self-scoped — it only decides whether a step's own failure fails the job, never whether anything downstream runs (that's a dependent's own `accept` set, see [Conditionals](https://fremvaerk.github.io/stroem/guides/conditionals/)). A flow-step `notify` depending on `build` would need `accept: [completed, failed]` (or `any`) just to run after a failed build — and if `build` also carried its own `continue_on_failure: true` so the job doesn't fail either, you've quietly turned a failed pipeline **green**, the opposite of a CI notification's job. A hook sidesteps both problems: it always fires exactly once on the real terminal outcome, and firing it never changes what that outcome is. That is why this page puts notification in hooks instead.
- **Webhook trigger**: External systems (GitHub) can trigger the pipeline via `POST /hooks/github-ci`
- **Webhook input**: `input.body.*` (e.g. `{{ input.body.ref }}`) accesses the parsed JSON body from the webhook request inside flow step templates — hook templates don't see it, only `hook.*` / `secret.*` (see [Hooks](https://fremvaerk.github.io/stroem/guides/hooks/))

## Setting up the GitHub webhook

1. Go to your GitHub repo → Settings → Webhooks → Add webhook
2. Set the Payload URL to `https://stroem.example.com/hooks/github-ci?secret=whsec_your_secret_here`
3. Set Content type to `application/json`
4. Select "Just the push event"

Now every push to the repository triggers the CI pipeline.

## Running manually

```bash
# Via API (simulating a webhook payload)
curl -X POST http://localhost:8080/api/workspaces/default/tasks/ci-pipeline/execute \
  -H "Content-Type: application/json" \
  -d '{"input": {"body": {"ref": "refs/heads/main"}}}'

# Via CLI
stroem trigger ci-pipeline --input '{"body": {"ref": "refs/heads/main"}}'
```

## Test matrix with `for_each`

Use [`for_each`](https://fremvaerk.github.io/stroem/guides/loops/) to run the same tests across multiple Node.js versions:

```yaml
actions:
  lint:
    type: script
    runner: docker
    script: |
      cd /workspace
      npm ci
      npm run lint

  test-version:
    type: docker
    image: "node:{{ input.node_version }}"
    cmd: |
      cd /app && npm ci && npm test &&
      echo "OUTPUT: {\"node\": \"{{ input.node_version }}\", \"passed\": true}"
    input:
      node_version: { type: string }

  build:
    type: script
    runner: docker
    script: |
      cd /workspace
      npm ci
      npm run build
      echo "OUTPUT: {\"artifact\": \"dist/app.tar.gz\"}"

  notify-ci:
    type: script
    script: |
      echo "CI result: {{ input.status }}"
    input:
      status: { type: string }

tasks:
  ci-matrix:
    mode: distributed
    input:
      body: { type: string }
    flow:
      lint:
        action: lint
      test:
        action: test-version
        for_each: ["18", "20", "22"]
        input:
          node_version: "{{ each.item }}"
      # lint and test run in parallel; build waits for both
      build:
        action: build
        depends_on: [lint, test]
    on_success:
      - action: notify-ci
        input:
          status: "success"
    on_error:
      - action: notify-ci
        input:
          status: "failed: {{ hook.error_message }}"
```

This creates `test[0]` (Node 18), `test[1]` (Node 20), `test[2]` (Node 22) — all running in parallel alongside lint. The `test-version` action uses `type: docker` to run each Node.js version's image directly — note that `type: docker` does **not** mount workspace files, so the application source must be baked into the image. The `lint` and `build` actions use `type: script` + `runner: docker` instead, which mounts the workspace at `/workspace`. Hooks only see the `hook.*` / `secret.*` template context, not individual step outputs — `hook.error_message` and `hook.failed_steps` cover the failure case; to notify with a step's own output, add a step to the flow instead of a hook.