Prometheus Metrics

Strøm exposes a Prometheus-compatible /metrics endpoint on every server replica.

Endpoint

• URL: https://<your-server>/metrics
• Method: GET
• Content-Type: text/plain; version=0.0.4
• Default auth: Authorization: Bearer <worker_token> (same token as /healthz/detail)
• Public mode: set metrics: { public: true } in server-config.yaml to disable auth

When running multiple replicas (see High Availability), every metric carries a replica_id label so per-pod series don’t collide.

When running with metrics.public: true, anyone who can reach the endpoint can read the replica_id global label, a UUID stable for the lifetime of each pod. Combined with network-level observability, this lets a scraper correlate metric series to specific pods. The UUID itself carries no secrets, but if pod identity is sensitive in your environment, keep metrics.public: false (the default) and authenticate scrapes with the worker token.

Metric Reference

Counters

Name	Labels	Description
stroem_http_requests_total	method, route, status	Requests served on /api/*. route uses the matched Axum pattern (e.g. /api/jobs/{id}), not raw URIs.
stroem_jobs_created_total	source_type	Jobs created. source_type is one of api, user, scheduler, webhook, event_source, hook, task, retry, rerun, restart.
stroem_jobs_completed_total	status	Jobs reaching terminal state. status is completed, failed, cancelled, or skipped.
stroem_input_file_uploads_total	origin, result	Input file uploads, counted once a request has passed its authentication and permission checks. origin is api (REST upload), mcp_inline (inline object in MCP execute_task) or mcp_url (/api/uploads/{token}). result is ok, too_large, invalid_name or error (error includes file storage is not configured). A request rejected before that — authentication, permission, a task with no file field, a missing name query, an unknown upload URL — is not counted. See File Inputs.
stroem_input_file_bytes_total	origin	Bytes of input files stored, counted for successful (ok) uploads only.

Histogram

Name	Labels	Buckets
stroem_http_request_duration_seconds	route	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds
stroem_snapshot_resolve_seconds	entry — claim, advance, init	0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1 seconds. Time to resolve the task+global state snapshot rows before rendering (claim, orchestration, creation). Pool-only; no archive access.

Gauges (sampled at scrape time)

Name	Labels	Source
stroem_leader_status	—	1 if this replica holds the HA leader lock, else 0
stroem_workers_active	—	Count of workers with status = 'active'
stroem_jobs_in_flight	status	Pending and running jobs (both buckets always emitted)
stroem_steps_ready	—	Claimable steps (status = 'ready', retry_at past or null); deferred steps (templated refs, resolved by the server) are not counted
stroem_background_task_alive	task	1 if the named loop is running. Labels: scheduler, recovery, event_source
stroem_background_task_last_tick_age_seconds	task	Seconds since the named loop’s last progress heartbeat (at the top of each iteration, at work-unit boundaries inside it, and before it sleeps). Absent until its first iteration. A loop can be alive yet hung — alert on this (e.g. > 300 for scheduler); /livez fails on the same condition

Workspace refresh

Name	Type	Labels	Meaning
stroem_workspace_peek_failures_total	counter	workspace	Workspace peeks that failed or timed out.
stroem_workspace_load_admission_skipped_total	counter	workspace, reason (busy, saturated)	Watcher loads not admitted.
stroem_workspace_last_successful_load_age_seconds	gauge	workspace	Seconds since a workspace last loaded successfully.
stroem_workspace_load_overdue	gauge	workspace	1 while a watcher load has exceeded its budget and still runs. Derived at scrape time, never stored.
stroem_workspace_load_permits_available	gauge	—	Free watcher-load permits on THIS replica. 0 also occurs with eight healthy loads in progress; alert only together with overdue > 0.

Alerting. Alert on stroem_workspace_load_permits_available == 0 and stroem_workspace_load_overdue > 0 for the same replica_id, sustained for a few minutes: every watcher slot on that replica is held by a load stuck in a phase that cannot be interrupted, and that replica has stopped refreshing workspaces. Neither signal fails /livez — a stale-but-serving server is healthy; recovering is an operator decision (restart the pod). stroem_workspace_last_successful_load_age_seconds growing past a few poll intervals means refreshes are being skipped — usually the remote is unreachable (see stroem_workspace_peek_failures_total).

Pin store

Pinned commits for git refs, held per replica (see pin_store).

Name	Type	Labels	Meaning
stroem_pin_loads_total	counter	workspace, result	Pinned-commit config loads (fetch + checkout + config load) that were not already cached, by outcome: ok, not_git, ref_not_found, commit_not_found, load_failed (the YAML at that commit does not load, or its sops/vals secrets have failed to load for an hour), unavailable (git server unreachable, load timeout, a sops/vals failure younger than an hour). A caller that waited behind another caller’s failed load and ran out of its own time budget also counts as unavailable. A sops/vals failure is remembered for 30 seconds; requests answered from that memory are not loads and are not counted. Tarball-only checkouts are not counted.
stroem_pins_cached	gauge	workspace	Pinned commit configs held in memory on THIS replica. 0 (not absent) for a git workspace with none cached.

A rising unavailable rate means claims are being released and pinned jobs are waiting for the git server (or for a commit’s secrets to decrypt); load_failed means a referenced commit has YAML that does not load, or secrets that have not decrypted for an hour, which fails every step and job that needs it.

Prometheus Scrape Config

Default (Bearer auth)

scrape_configs:
  - job_name: stroem
    metrics_path: /metrics
    scheme: https
    bearer_token_file: /var/run/secrets/stroem/worker-token
    static_configs:
      - targets: ['stroem.internal:443']

Public mode (no auth)

scrape_configs:
  - job_name: stroem
    metrics_path: /metrics
    static_configs:
      - targets: ['stroem.internal:80']

Helm + Prometheus Operator

Set in values.yaml:

serviceMonitor:
  enabled: true
  bearerTokenSecret:
    name: stroem-worker-token  # the secret backing STROEM__WORKER_TOKEN
    key: token

Starter Alerts

groups:
  - name: stroem
    rules:
      - alert: StroemNoLeader
        expr: max(stroem_leader_status) == 0
        for: 1m
        annotations:
          summary: "No Strøm replica is the HA leader"

      - alert: StroemNoActiveWorkers
        expr: max(stroem_workers_active) < 1
        for: 2m
        annotations:
          summary: "No active Strøm workers — jobs will not be claimed"

      - alert: StroemQueueGrowing
        # `stroem_steps_ready` is a gauge — use `delta()` not `rate()`.
        # Fires when ready-step count grew by >0 over 10m, indicating
        # workers can't keep up with new step arrivals.
        expr: delta(stroem_steps_ready[10m]) > 0 and stroem_steps_ready > 10
        for: 10m
        annotations:
          summary: "Strøm step queue growing — workers not keeping up"

Multi-replica note: stroem_steps_ready is sampled at scrape time from the same shared Postgres for every replica, so all replicas should observe the same value. However, delta() over a per-replica time series can show artifacts if a replica restarts mid-window (the new replica_id starts a fresh time series). For HA deployments, prefer delta(max(stroem_steps_ready) without (replica_id) [10m]) to aggregate before computing the delta.

Notes

• Gauges that fail to sample (DB timeout, transient error) are not fabricated as zero — the sample is skipped, which Prometheus correctly treats as stale. Don’t write alerts that assume “missing means zero”.
• Per-workspace and per-task labels are deliberately not present in v1 (cardinality risk). Future opt-in is on the roadmap.
• /worker/* polling traffic and /hooks/* are deliberately excluded from stroem_http_requests_total to keep RED signals focused on user-facing API latency.