Orchestration

A build is more than a DAG that runs once and exits. A real deployment pipeline needs to pause for a human approval or a soak period and pick back up — possibly days later, in a different process — fan a step out over a runtime list of services or tenants, and cancel cleanly, undoing exactly the work it did. Zuke builds all three on the same primitive: a durable run whose state survives the process that started it.

Suspend & resume

A target declared with .waitsFor() is a bodyless gate: reached in the graph, it checks whether its trigger is already satisfied. If so, it passes through and dependents run in the same process. If not, the run's state is saved, the run is marked suspended, any independent branches finish, and the process exits 0 — a suspended run has not failed. A later zuke resume, possibly in a different process, re-instantiates the build and continues only the targets that haven't already succeeded. This requires a state store (auto-enables the .zuke/runs filesystem store).

// TargetBuilder:  waitsFor(configure: Configure<WaitSettings>): this
// WaitSettings (fluent):
on(trigger: WaitTrigger): this
timeout(duration: string | number): this          // "72h" or ms
onTimeout(disposition: OnTimeout): this            // default "fail"
export type OnTimeout = () => TargetBuilder | "fail" | "cancel-run";

// Triggers:
export function externalSignal(name: string): WaitTrigger
export function resumeWhen(check: () => boolean | Promise<boolean>, options?: { interval?: string | number }): WaitTrigger
// …and from @zuke/gh, built on the same exported WaitTrigger / WaitContext seam:
export function githubWorkflow(configure: Configure<GithubWorkflowSettings>): WaitTrigger

Two built-in triggers: externalSignal(name) waits for a named signal delivered from outside the run (an approval, a webhook); resumeWhen(check) waits for a predicate to become true, re-evaluated on demand rather than polled continuously. A delivered signal's payload shows up on ctx.signals as { data, receivedAt }.

.timeout() bounds how long a wait may stay suspended; .onTimeout() decides what happens when it elapses — "fail" (the default), "cancel-run", or a thunk returning a target to run instead (a rollback). The canonical shape — deploy, wait for an external approval, then promote:

import { Build, externalSignal, target } from "jsr:@zuke/core";

class Deploy extends Build {
  deployToSit = target().executes(async (ctx) => {
    await applyToSit();
    await ctx.state.set({ at: "sit-7" });   // survives the suspend
  });

  awaitTesting = target()
    .dependsOn(this.deployToSit)
    .waitsFor((s) => s.on(externalSignal("testing-approved")).timeout("72h").onTimeout(() => this.rollback));

  promoteToProd = target()
    .dependsOn(this.awaitTesting)
    .executes((ctx) => {
      const approval = ctx.signals.get("testing-approved");   // payload
      promote(approval?.data);
    });

  rollback = target().executes(() => rollBackSit());
}

Resuming it once QA signs off:

zuke resume <run-id> --signal testing-approved --data '{"by":"qa"}'   # deliver a signal
zuke resume --check [<run-id>]                                        # reap abandoned runs, then re-check predicate/timeout across ALL suspended runs
# also: --force-graph, --resume-degraded, --actor <name>

Resume is exactly-once: it's a compare-and-swap from suspended to running, so one resumer wins and every other — a retrying cron, a doubled webhook — gets a clean AlreadyResumedError and exits non-zero instead of running the promotion twice. Resume also re-checks that the build's graph still matches the suspended run; an added, removed, or re-wired target is a hard error pointing at --force-graph to override it deliberately. resumeWhen doesn't self-poll — Zuke re-evaluates the predicate (and any elapsed timeout) on each --check, so one sweeping cron enforces every deadline across every suspended run.

Waiting on an external GitHub workflow

@zuke/gh's githubWorkflow trigger dispatches a GitHub Actions workflow — often in another repo — and suspends until it finishes, replacing hand-rolled "dispatch, then poll gh run list" glue. It is a third-party trigger built on the exported WaitTrigger / WaitContext seam, so you can write your own the same way.

import { Build, run, target } from "jsr:@zuke/core";
import { githubWorkflow, readWorkflowResult } from "jsr:@zuke/gh";

class Release extends Build {
  e2e = target().waitsFor((s) =>
    s.on(githubWorkflow((g) => g.repo("acme/app").workflow("e2e.yml").ref("main")))
      .timeout("2h").onTimeout(() => this.rollback)
  );

  ship = target().dependsOn(this.e2e).executes((ctx) => {
    // The gate publishes its result to its own state — read it via stateOf.
    const result = readWorkflowResult(ctx.stateOf("e2e"));
    if (!result?.passed) throw new Error("e2e suite failed");
  });

  rollback = target().executes(() => rollBack());
}

• Dispatch-once, then poll. On first reach it records a correlation marker in the gate's durable state, dispatches, and suspends. Each resume --check polls the run; a resume in a different process never re-dispatches, because the marker persisted with the run. The marker is written before the dispatch call, so a process killed mid-dispatch re-polls rather than starting a second workflow — the cost of that ordering being the mirror case, where a dispatch that never landed is reported when the discovery window elapses instead of immediately.
• Correlation. workflow_dispatch returns no run id, so by default the trigger passes a marker input (zuke_marker) and matches it against the run's display title. For a workflow you can't modify, .correlate("created-window") correlates best-effort: it claims the workflow_dispatch run on the dispatch ref created just after dispatch, and fails loudly if two candidates share the window.
• Fast-fail. If no run is identified within a short discovery window (.discoveryTimeout(...), default one minute), the gate fails with guidance instead of eating the whole .timeout(). The deadline is measured from the persisted dispatch time, so it holds across a suspend/resume.
• Result. On completion the per-job conclusions ({ passed, jobs: [{ name, conclusion, url }] }) are written to the gate target's state; a dependent reads them with readWorkflowResult(ctx.stateOf("<gate>")) and branches on a failed suite.
• Auth uses GH_TOKEN / GITHUB_TOKEN; the GitHub API is an injectable transport, so it is testable without a real GitHub.

The dispatched workflow's contract

The gate is only half the wiring. Three requirements on the target workflow are load-bearing — each is a deterministic dispatch 422 or a gate that hangs until timeout:

# .github/workflows/e2e.yml — in the repo being dispatched
on:
  workflow_dispatch:
    inputs:
      zuke_marker: { required: false }   # rename → .markerInput("name") on the gate
      # any `required: true` input here must be supplied via .inputs(...) on the gate
run-name: ${{ inputs.zuke_marker }}      # the ENTIRE run-name; equality, not substring

• Marker input name. GitHub 422s a dispatch carrying an input the workflow does not declare, so a workflow that names its marker input anything else rejects the dispatch outright — declare zuke_marker, or point the gate at your name with .markerInput("<name>").
• Required inputs. Every required: true input on the target workflow must be supplied from the gate with .inputs({ … }) / .input(name, value), or the dispatch 422s. The settings lambda is evaluated when the build is defined and has no access to run state — it can read params but not a value an earlier target recorded, so an input produced at run time needs a custom WaitTrigger.
• Strict run-name equality. Marker correlation matches display_title === marker — exact equality, not substring. A decorated run-name like run-name: E2E [${{ inputs.zuke_marker }}] dispatches fine and then never correlates, so the gate just times out. Echo the marker as the workflow's entire run-name:, or switch to .correlate("created-window").

Abandoned runs — reaping

A run that ends cleanly settles itself, and a run that suspends is picked up by the sweep above. A run whose process was killed does neither: it stays running, and a resume only ever acts on suspended runs. Anything that run recorded as owed — a crash-durable effect above all — would wait forever.

So zuke resume --check looks at running runs first, before it sweeps the suspended ones:

• Is anyone still there? — asked first, always. The run's lease answers it: a live process keeps renewing, so a lease that can be acquired means its holder is gone; one that cannot means the run is merely slow, and slow is left alone. A live run is never settled or moved, deadline or no deadline, because doing so would run its compensations beside the work they undo.
• Nobody there → the run goes back to suspended, with a reap event on its audit trail saying why. The same sweep then resumes it, so an abandoned run is recovered in one pass rather than waiting another interval for nothing but a state change.
• Nobody there, and past its deadline() → the run is settled failed, compensations and all. It did not stop because anyone asked; it ran out of time, and anything waiting on it needs an answer rather than silence.
• Only this build's runs. A state store is commonly shared and a listing has no build filter, so the sweep skips runs it does not recognise: the record's build name and its root target must both be this build's. Before settling a run — irreversible, and it runs the compensations — the graph has to agree as well, because a class name is not an identity. Handing a run back to suspended asks only the looser question, since that is reversible and a resume refuses a graph it does not recognise.

A run left cancelling by a settlement whose own process died is finished too, in whichever terminal that settlement was heading for — recorded on the run (intendedTerminal), because the process that finishes it is not the one that began it.

Build.deadline()

A wall-clock budget for a whole run. It bounds running, not existing:

class Ci extends Build {
  override deadline() {
    return "45m";
  }
}

A run parked at a .waitsFor(...) gate is not spending it: the deadline is pushed forward on resume by however long the run was parked, so a build with a 72-hour approval gate and a 45-minute deadline still has its 45 minutes of running time when the approval finally arrives. The gate's own .timeout() is what bounds the waiting.

A live run is never settled for time either — the sweep asks whether anyone is still working on the run before it looks at the deadline, and leaves it alone if someone is. That costs nothing the deadline was for: a process that hangs stops renewing its lease, and a run killed and abandoned repeatedly has no holder at all. What it rules out is settling a run underneath a process that is still working on it.

What it is really for is the run that stops making progress without failing: a process that hangs, or one killed so hard that its work is picked up and abandoned repeatedly. Without a deadline such a run has no end state at all; with one it reaches a terminal status, which is what anything downstream is waiting for. The value is stamped once at creation as deadlineAt on the run record.

Fan-out

.forEach() runs the same ordered pipeline over a runtime list — deploy N repos, migrate N tenants — with bounded concurrency and per-item failure isolation. Items run concurrently up to concurrency; within one item, its stages still run sequentially, each depending on the one before — there's no barrier between items, so a fast item can finish its whole pipeline while a slow one is still on its first stage.

forEach<Item>(items: () => readonly Item[], factory: ForEachFactory<Item>, configure?: Configure<ForEachSettings>): this
export type ForEachFactory<Item> = (item: Item, index: number) => Record<string, TargetBuilder>;  // ordered stages; each depends on the previous

// ForEachSettings:
concurrency(limit: number): this            // clamped ≥1; default = host CPU count
continueOnItemFailure(on?: boolean): this

The canonical batch deploy — one three-stage pipeline per repo:

import { Build, parameter, target } from "jsr:@zuke/core";

class CD extends Build {
  repos = parameter("services to deploy").required().array();

  deployBatch = target().forEach(
    () => this.repos.value,                    // items thunk (string[])
    (repo) => ({                               // ordered pipeline per item
      checks: target().executes(() => checkDeployable(repo)),
      fork:   target().executes(() => forkImage(repo)),
      deploy: target().executes((ctx) => applyToSit(repo, ctx)),
    }),
    (s) => s.concurrency(3).continueOnItemFailure(),
  );
}

concurrency clamps to at least 1 and defaults to the host's CPU count. continueOnItemFailure() decides whether a failed item's siblings keep going: without it, the first item failure stops the batch; with it, a failed stage skips that item's later stages but lets other items finish. Either way the fan-out target itself fails if any item failed — add .proceedAfterFailure() if partial success should still be a passing build.

Each item is first-class for reporting: sub-targets are materialised at run time and named parent[<item>].<stage> (e.g. deployBatch[api].deploy), and each gets its own summary row and its own entry in the run record — zuke runs show gives per-item verdicts, not just a batch total. Static views (zuke --list, zuke graph) show only the single fan-out node, annotated [fan-out], since the item list isn't known until the run starts.

An .onCancel() on a fan-out sub-target runs too. Cancel re-materialises the fan-out, matches each item's sub-targets by name against the record, and runs the compensation of every item that had succeeded — or was still in-flight when the cancel landed (a mid-deploy item has partial work to undo). Items unwind before the parent's own compensation, in reverse order; a throwing item compensation is recorded and the walk continues, exactly as for an ordinary target. Each item's cleanup reads that item's own persisted state (ctx.state), and lands its own compensate entry in the audit trail (naming the runtime sub-target, e.g. deployBatch[repo-a].deploy) alongside the summary.

Nested fan-out works too: a .forEach() stage that is itself a .forEach() has its inner items' .onCancel() run, matched by the same nested parent[item].stage[inner].stage names.

Every per-item stage is a full target: it takes ctx, can read and write ctx.state, and can carry its own .timeout() and .retry().

Cancellation & compensation

Cancelling should be safe and complete: work undone, locks released, the run's record ending cancelled. A compensation target, registered with .onCancel(), runs if and only if its target succeeded, walked in reverse topological order — later work is unwound before what it was built on. This requires a state store (auto-enabled, same as suspend/resume).

onCancel(compensation: OnCancel): this
export type OnCancel = TargetBuilder | (() => TargetBuilder);  // thunk references a target declared below

Trigger a cancel from any of three places:

zuke cancel <run-id>            # cancel a persisted run from any process (accepts --actor <name>)
# Ctrl-C / SIGTERM             # cancels the run in the current process
# MCP cancel_run tool          # cancel over MCP (gated like a run tool)

class CD extends Build {
  deploy = target()
    .executes((ctx) => ctx.state.set({ slot: "sit-7" }))   // record what it did
    .onCancel(() => this.rollback);                          // …and how to undo it

  rollback = target().executes((ctx) => tearDown(ctx.state.get().slot));   // reads deploy's meta
}

The compensation body gets a normal ctx, but its ctx.state exposes the original target's persisted metadata — not its own — so rollback above reads exactly what deploy recorded at work time (ctx.state.get().slot). Persist whatever a rollback will need while the original target is still running; there's nothing left to compute it from afterward.

Lazy ordering — override orderWith(targets)

Same shape as declared ordering edges, but async and resolved per run: use it when the ordering must be loaded at run time — read a monorepo's dependency-graph.json, hit an API — rather than declared statically. Return the same [before, after] OrderingEdge pairs; they merge with any extraEdges, and share its rules (endpoints outside the run's set are ignored, cycles reported). Both are honoured by a run and by zuke cancel's reverse-order compensation walk, but — being resolved only when a run plans — neither shows in the static graph / --list views nor in generated CI.

override async orderWith(t: Map<string, Target>): Promise<OrderingEdge[]> {
  const graph = await loadDependencyGraph(); // read it at run time
  return graph.edges.flatMap(([before, after]) => {
    const from = t.get(before), to = t.get(after);
    return from && to ? [[from, to] as OrderingEdge] : [];
  });
}

Parameters for fan-out

.array() turns a parameter into a comma-separated or repeatable list — the natural feed for a forEach items thunk. It composes last, after any kind or .options() already declared, so every element is validated before any target runs:

tags     = parameter("Image tags").array();                            // string[]
workers  = parameter("Worker ids").number().array();                   // number[]; "1,x" rejected
services = parameter("Services").options("api","web","worker").array(); // each element validated
repos    = parameter("services to deploy").required().array();         // required list (.required() BEFORE .array()); feeds forEach items thunk

./zuke deploy --tags latest,canary          # ["latest","canary"]
./zuke deploy --tags latest --tags canary   # same (repeat == comma)
TAGS=latest,canary ./zuke deploy            # from env

Blank entries are dropped and an unsupplied list defaults to []. Read it inside the forEach items thunk with this.<name>.value, as repos does in the batch deploy example above.